  Recent works show that speech separation guided diarization (SSGD) is an
increasingly promising direction, mainly thanks to the recent progress in
speech separation. It performs diarization by first separating the speakers and
then applying voice activity detection (VAD) on each separated stream. In this
work we conduct an in-depth study of SSGD in the conversational telephone
speech (CTS) domain, focusing mainly on low-latency streaming diarization
applications. We consider three state-of-the-art speech separation (SSep)
algorithms and study their performance both in online and offline scenarios,
considering non-causal and causal implementations as well as continuous SSep
(CSS) windowed inference. We compare different SSGD algorithms on two widely
used CTS datasets: CALLHOME and Fisher Corpus (Part 1 and 2) and evaluate both
separation and diarization performance. To improve performance, a novel, causal
and computationally efficient leakage removal algorithm is proposed, which
significantly decreases false alarms. We also explore, for the first time,
fully end-to-end SSGD integration between SSep and VAD modules. Crucially, this
enables fine-tuning on real-world data for which oracle speakers sources are
not available. In particular, our best model achieves 8.8% DER on CALLHOME,
which outperforms the current state-of-the-art end-to-end neural diarization
model, despite being trained on an order of magnitude less data and having
significantly lower latency, i.e., 0.1 vs. 1 seconds. Finally, we also show
that the separated signals can be readily used also for automatic speech
recognition, reaching performance close to using oracle sources in some
configurations.
