  Attention-based methods and Connectionist Temporal Classification (CTC)
network have been promising research directions for end-to-end (E2E) Automatic
Speech Recognition (ASR). The joint CTC/Attention model has achieved great
success by utilizing both architectures during multi-task training and joint
decoding. In this work, we present a multi-stream framework based on joint
CTC/Attention E2E ASR with parallel streams represented by separate encoders
aiming to capture diverse information. On top of the regular attention
networks, the Hierarchical Attention Network (HAN) is introduced to steer the
decoder toward the most informative encoders. A separate CTC network is
assigned to each stream to force monotonic alignments. Two representative
framework have been proposed and discussed, which are Multi-Encoder
Multi-Resolution (MEM-Res) framework and Multi-Encoder Multi-Array (MEM-Array)
framework, respectively. In MEM-Res framework, two heterogeneous encoders with
different architectures, temporal resolutions and separate CTC networks work in
parallel to extract complimentary information from same acoustics. Experiments
are conducted on Wall Street Journal (WSJ) and CHiME-4, resulting in relative
Word Error Rate (WER) reduction of 18.0-32.1% and the best WER of 3.6% in the
WSJ eval92 test set. The MEM-Array framework aims at improving the far-field
ASR robustness using multiple microphone arrays which are activated by separate
encoders. Compared with the best single-array results, the proposed framework
has achieved relative WER reduction of 3.7% and 9.7% in AMI and DIRHA
multi-array corpora, respectively, which also outperforms conventional fusion
strategies.
