  Target speech separation refers to extracting a target speaker's voice from
an overlapped audio of simultaneous talkers. Previously the use of visual
modality for target speech separation has demonstrated great potentials. This
work proposes a general multi-modal framework for target speech separation by
utilizing all the available information of the target speaker, including
his/her spatial location, voice characteristics and lip movements. Also, under
this framework, we investigate on the fusion methods for multi-modal joint
modeling. A factorized attention-based fusion method is proposed to aggregate
the high-level semantic information of multi-modalities at embedding level.
This method firstly factorizes the mixture audio into a set of acoustic
subspaces, then leverages the target's information from other modalities to
enhance these subspace acoustic embeddings with a learnable attention scheme.
To validate the robustness of proposed multi-modal separation model in
practical scenarios, the system was evaluated under the condition that one of
the modalities is temporarily missing, invalid or corrupted. Experiments are
conducted on a large-scale audio-visual dataset collected from YouTube (to be
released) that spatialized by simulated room impulse responses (RIRs).
Experiment results illustrate that our proposed multi-modal framework
significantly outperforms single-modal and bi-modal speech separation
approaches, while can still support real-time processing.
