  In this paper, we propose an efficient and accurate streaming speech
recognition model based on the FastConformer architecture. We adapted the
FastConformer architecture for streaming applications through: (1) constraining
both the look-ahead and past contexts in the encoder, and (2) introducing an
activation caching mechanism to enable the non-autoregressive encoder to
operate autoregressively during inference. The proposed model is thoughtfully
designed in a way to eliminate the accuracy disparity between the train and
inference time which is common for many streaming models. Furthermore, our
proposed encoder works with various decoder configurations including
Connectionist Temporal Classification (CTC) and RNN-Transducer (RNNT) decoders.
Additionally, we introduced a hybrid CTC/RNNT architecture which utilizes a
shared encoder with both a CTC and RNNT decoder to boost the accuracy and save
computation. We evaluate the proposed model on LibriSpeech dataset and a
multi-domain large scale dataset and demonstrate that it can achieve better
accuracy with lower latency and inference time compared to a conventional
buffered streaming model baseline. We also showed that training a model with
multiple latencies can achieve better accuracy than single latency models while
it enables us to support multiple latencies with a single model. Our
experiments also showed the hybrid architecture would not only speedup the
convergence of the CTC decoder but also improves the accuracy of streaming
models compared to single decoder models.
