  Audio-visual learning has demonstrated promising results in many classical
speech tasks (e.g., speech separation, automatic speech recognition, wake-word
spotting). We believe that introducing visual modality will also benefit
speaker diarization. To date, target-speaker voice activity detection (TS-VAD)
plays an essential role in highly accurate speaker diarization. However,
previous TS-VAD models take audio features and utilize the speaker's acoustic
footprint to distinguish his or her personal speech activities, which is
susceptible to overlapped speaking in multi-speaker scenarios. Although visual
information naturally tolerates overlapped speech, it easily suffers from
spatial occlusion. The potential modality-missing problem blocks TS-VAD towards
an audio-visual approach.
  This paper proposes a multi-input multi-output target-speaker voice activity
detection (MIMO-TSVAD) framework for speaker diarization. The proposed method
can take audio-visual input and leverage the speaker's acoustic footprint or
lip track to flexibly conduct audio-based, video-based, and audio-visual
speaker diarization in a unified sequence-to-sequence architecture.
Experimental results show that the MIMO-TSVAD framework demonstrates
state-of-the-art performance on the VoxConverse, DIHARD-III, and MISP 2022
datasets under corresponding evaluation metrics, obtaining the diarization
error rates (DERs) of 4.18%, 10.10%, and 8.15%, respectively. In addition, it
can perform robustly in heavy lip-missing scenarios.
