  The joint training framework for speech enhancement and recognition methods
have obtained quite good performances for robust end-to-end automatic speech
recognition (ASR). However, these methods only utilize the enhanced feature as
the input of the speech recognition component, which are affected by the speech
distortion problem. In order to address this problem, this paper proposes a
gated recurrent fusion (GRF) method with joint training framework for robust
end-to-end ASR. The GRF algorithm is used to dynamically combine the noisy and
enhanced features. Therefore, the GRF can not only remove the noise signals
from the enhanced features, but also learn the raw fine structures from the
noisy features so that it can alleviate the speech distortion. The proposed
method consists of speech enhancement, GRF and speech recognition. Firstly, the
mask based speech enhancement network is applied to enhance the input speech.
Secondly, the GRF is applied to address the speech distortion problem. Thirdly,
to improve the performance of ASR, the state-of-the-art speech transformer
algorithm is used as the speech recognition component. Finally, the joint
training framework is utilized to optimize these three components,
simultaneously. Our experiments are conducted on an open-source Mandarin speech
corpus called AISHELL-1. Experimental results show that the proposed method
achieves the relative character error rate (CER) reduction of 10.04\% over the
conventional joint enhancement and transformer method only using the enhanced
features. Especially for the low signal-to-noise ratio (0 dB), our proposed
method can achieves better performances with 12.67\% CER reduction, which
suggests the potential of our proposed method.
