  Speaker extraction aims to mimic humans' selective auditory attention by
extracting a target speaker's voice from a multi-talker environment. It is
common to perform the extraction in frequency-domain, and reconstruct the
time-domain signal from the extracted magnitude and estimated phase spectra.
However, such an approach is adversely affected by the inherent difficulty of
phase estimation. Inspired by Conv-TasNet, we propose a time-domain speaker
extraction network (SpEx) that converts the mixture speech into multi-scale
embedding coefficients instead of decomposing the speech signal into magnitude
and phase spectra. In this way, we avoid phase estimation. The SpEx network
consists of four network components, namely speaker encoder, speech encoder,
speaker extractor, and speech decoder. Specifically, the speech encoder
converts the mixture speech into multi-scale embedding coefficients, the
speaker encoder learns to represent the target speaker with a speaker
embedding. The speaker extractor takes the multi-scale embedding coefficients
and target speaker embedding as input and estimates a receptive mask. Finally,
the speech decoder reconstructs the target speaker's speech from the masked
embedding coefficients. We also propose a multi-task learning framework and a
multi-scale embedding implementation. Experimental results show that the
proposed SpEx achieves 37.3%, 37.7% and 15.0% relative improvements over the
best baseline in terms of signal-to-distortion ratio (SDR), scale-invariant SDR
(SI-SDR), and perceptual evaluation of speech quality (PESQ) under an open
evaluation condition.
