  Recently, end-to-end models have been widely used in automatic speech
recognition (ASR) systems. Two of the most representative approaches are
connectionist temporal classification (CTC) and attention-based encoder-decoder
(AED) models. Autoregressive transformers, variants of AED, adopt an
autoregressive mechanism for token generation and thus are relatively slow
during inference. In this paper, we present a comprehensive study of a CTC
Alignment-based Single-Step Non-Autoregressive Transformer (CASS-NAT) for
end-to-end ASR. In CASS-NAT, word embeddings in the autoregressive transformer
(AT) are substituted with token-level acoustic embeddings (TAE) that are
extracted from encoder outputs with the acoustical boundary information offered
by the CTC alignment. TAE can be obtained in parallel, resulting in a parallel
generation of output tokens. During training, Viterbi-alignment is used for TAE
generation, and multiple training strategies are further explored to improve
the word error rate (WER) performance. During inference, an error-based
alignment sampling method is investigated in depth to reduce the alignment
mismatch in the training and testing processes. Experimental results show that
the CASS-NAT has a WER that is close to AT on various ASR tasks, while
providing a ~24x inference speedup. With and without self-supervised learning,
we achieve new state-of-the-art results for non-autoregressive models on
several datasets. We also analyze the behavior of the CASS-NAT decoder to
explain why it can perform similarly to AT. We find that TAEs have similar
functionality to word embeddings for grammatical structures, which might
indicate the possibility of learning some semantic information from TAEs
without a language model.
