  The performance of acoustic models degrades notably in noisy environments.
Speech enhancement (SE) can be used as a front-end strategy to aid automatic
speech recognition (ASR) systems. However, existing training objectives of SE
methods are not fully effective at integrating speech-text and noisy-clean
paired data for training toward unseen ASR systems. In this study, we propose a
general denoising framework, D4AM, for various downstream acoustic models. Our
framework fine-tunes the SE model with the backward gradient according to a
specific acoustic model and the corresponding classification objective. In
addition, our method aims to consider the regression objective as an auxiliary
loss to make the SE model generalize to other unseen acoustic models. To
jointly train an SE unit with regression and classification objectives, D4AM
uses an adjustment scheme to directly estimate suitable weighting coefficients
rather than undergoing a grid search process with additional training costs.
The adjustment scheme consists of two parts: gradient calibration and
regression objective weighting. The experimental results show that D4AM can
consistently and effectively provide improvements to various unseen acoustic
models and outperforms other combination setups. Specifically, when evaluated
on the Google ASR API with real noisy data completely unseen during SE
training, D4AM achieves a relative WER reduction of 24.65% compared with the
direct feeding of noisy input. To our knowledge, this is the first work that
deploys an effective combination scheme of regression (denoising) and
classification (ASR) objectives to derive a general pre-processor applicable to
various unseen ASR systems. Our code is available at
https://github.com/ChangLee0903/D4AM.
