  For fine-grained generation and recognition tasks such as
minimally-supervised text-to-speech (TTS), voice conversion (VC), and automatic
speech recognition (ASR), the intermediate representations extracted from
speech should serve as a "bridge" between text and acoustic information,
containing information from both modalities. The semantic content is
emphasized, while the paralinguistic information such as speaker identity and
acoustic details should be de-emphasized. However, existing methods for
extracting fine-grained intermediate representations from speech suffer from
issues of excessive redundancy and dimension explosion. Contrastive learning is
a good method for modeling intermediate representations from two modalities.
However, existing contrastive learning methods in the audio field focus on
extracting global descriptive information for downstream audio classification
tasks, making them unsuitable for TTS, VC, and ASR tasks. To address these
issues, we propose a method named "Contrastive Token-Acoustic Pretraining
(CTAP)", which uses two encoders to bring phoneme and speech into a joint
multimodal space, learning how to connect phoneme and speech at the frame
level. The CTAP model is trained on 210k speech and phoneme pairs, achieving
minimally-supervised TTS, VC, and ASR. The proposed CTAP method offers a
promising solution for fine-grained generation and recognition downstream tasks
in speech processing. We provide a website with audio samples.
