  Continuous pseudo-labeling (PL) algorithms such as slimIPL have recently
emerged as a powerful strategy for semi-supervised learning in speech
recognition. In contrast with earlier strategies that alternated between
training a model and generating pseudo-labels (PLs) with it, here PLs are
generated in end-to-end manner as training proceeds, improving training speed
and the accuracy of the final model. PL shares a common theme with
teacher-student models such as distillation in that a teacher model generates
targets that need to be mimicked by the student model being trained. However,
interestingly, PL strategies in general use hard-labels, whereas distillation
uses the distribution over labels as the target to mimic. Inspired by
distillation we expect that specifying the whole distribution (aka soft-labels)
over sequences as the target for unlabeled data, instead of a single best pass
pseudo-labeled transcript (hard-labels) should improve PL performance and
convergence. Surprisingly and unexpectedly, we find that soft-labels targets
can lead to training divergence, with the model collapsing to a degenerate
token distribution per frame. We hypothesize that the reason this does not
happen with hard-labels is that training loss on hard-labels imposes
sequence-level consistency that keeps the model from collapsing to the
degenerate solution. In this paper, we show several experiments that support
this hypothesis, and experiment with several regularization approaches that can
ameliorate the degenerate collapse when using soft-labels. These approaches can
bring the accuracy of soft-labels closer to that of hard-labels, and while they
are unable to outperform them yet, they serve as a useful framework for further
improvements.
