  This paper describes a dataset and protocols for evaluating continuous speech
separation algorithms. Most prior studies on speech separation use
pre-segmented signals of artificially mixed speech utterances which are mostly
\emph{fully} overlapped, and the algorithms are evaluated based on
signal-to-distortion ratio or similar performance metrics. However, in natural
conversations, a speech signal is continuous, containing both overlapped and
overlap-free components. In addition, the signal-based metrics have very weak
correlations with automatic speech recognition (ASR) accuracy. We think that
not only does this make it hard to assess the practical relevance of the tested
algorithms, it also hinders researchers from developing systems that can be
readily applied to real scenarios. In this paper, we define continuous speech
separation (CSS) as a task of generating a set of non-overlapped speech signals
from a \textit{continuous} audio stream that contains multiple utterances that
are \emph{partially} overlapped by a varying degree. A new real recorded
dataset, called LibriCSS, is derived from LibriSpeech by concatenating the
corpus utterances to simulate a conversation and capturing the audio replays
with far-field microphones. A Kaldi-based ASR evaluation protocol is also
established by using a well-trained multi-conditional acoustic model. By using
this dataset, several aspects of a recently proposed speaker-independent CSS
algorithm are investigated. The dataset and evaluation scripts are available to
facilitate the research in this direction.
