  As one of the major branches of automatic speech recognition, attention-based
models greatly improves the feature representation ability of the model. In
particular, the multi-head mechanism is employed in the attention, hoping to
learn speech features of more aspects in different attention subspaces. For
speech recognition of complex languages, on the one hand, a small head size
will lead to an obvious shortage of learnable aspects. On the other hand, we
need to reduce the dimension of each subspace to keep the size of the overall
feature space unchanged when we increase the number of heads, which will
significantly weaken the ability to represent the feature of each subspace.
Therefore, this paper explores how to use a small attention subspace to
represent complete speech features while ensuring many heads. In this work we
propose a novel neural network architecture, namely, pyramid multi-branch
fusion DCNN with multi-head self-attention. The proposed architecture is
inspired by Dilated Convolution Neural Networks (DCNN), it uses multiple
branches with DCNN to extract the feature of the input speech under different
receptive fields. To reduce the number of parameters, every two branches are
merged until all the branches are merged into one. Thus, its shape is like a
pyramid rotated 90 degrees. We demonstrate that on Aishell-1, a widely used
Mandarin speech dataset, our model achieves a character error rate (CER) of
6.45% on the test sets.
