  Multilingual ASR technology simplifies model training and deployment, but its
accuracy is known to depend on the availability of language information at
runtime. Since language identity is seldom known beforehand in real-world
scenarios, it must be inferred on-the-fly with minimum latency. Furthermore, in
voice-activated smart assistant systems, language identity is also required for
downstream processing of ASR output. In this paper, we introduce streaming,
end-to-end, bilingual systems that perform both ASR and language identification
(LID) using the recurrent neural network transducer (RNN-T) architecture. On
the input side, embeddings from pretrained acoustic-only LID classifiers are
used to guide RNN-T training and inference, while on the output side, language
targets are jointly modeled with ASR targets. The proposed method is applied to
two language pairs: English-Spanish as spoken in the United States, and
English-Hindi as spoken in India. Experiments show that for English-Spanish,
the bilingual joint ASR-LID architecture matches monolingual ASR and
acoustic-only LID accuracies. For the more challenging (owing to
within-utterance code switching) case of English-Hindi, English ASR and LID
metrics show degradation. Overall, in scenarios where users switch dynamically
between languages, the proposed architecture offers a promising simplification
over running multiple monolingual ASR models and an LID classifier in parallel.
