  Self-supervised speech pre-training empowers the model with the contextual
structure inherent in the speech signal while self-supervised text pre-training
empowers the model with linguistic information. Both of them are beneficial for
downstream speech tasks such as ASR. However, the distinct pre-training
objectives make it challenging to jointly optimize the speech and text
representation in the same model. To solve this problem, we propose
Text-Enhanced Self-Supervised Speech Pre-training (TESSP), aiming to
incorporate the linguistic information into speech pre-training. Our model
consists of three parts, i.e., a speech encoder, a text encoder and a shared
encoder. The model takes unsupervised speech and text data as the input and
leverages the common HuBERT and MLM losses respectively. We also propose
phoneme up-sampling and representation swapping to enable joint modeling of the
speech and text information. Specifically, to fix the length mismatching
problem between speech and text data, we phonemize the text sequence and
up-sample the phonemes with the alignment information extracted from a small
set of supervised data. Moreover, to close the gap between the learned speech
and text representations, we swap the text representation with the speech
representation extracted by the respective private encoders according to the
alignment information. Experiments on the Librispeech dataset shows the
proposed TESSP model achieves more than 10% improvement compared with WavLM on
the test-clean and test-other sets. We also evaluate our model on the SUPERB
benchmark, showing our model has better performance on Phoneme Recognition,
Acoustic Speech Recognition and Speech Translation compared with WavLM.
