  Recently, there has been a growing interest in text-to-speech (TTS) methods
that can be trained with minimal supervision by combining two types of discrete
speech representations and using two sequence-to-sequence tasks to decouple
TTS. However, existing methods suffer from three problems: the high
dimensionality and waveform distortion of discrete speech representations, the
prosodic averaging problem caused by the duration prediction model in
non-autoregressive frameworks, and the information redundancy and dimension
explosion problems of existing semantic encoding methods. To address these
problems, three progressive methods are proposed. First, we propose
Diff-LM-Speech, an autoregressive structure consisting of a language model and
diffusion models, which models the semantic embedding into the mel-spectrogram
based on a diffusion model to achieve higher audio quality. We also introduce a
prompt encoder structure based on a variational autoencoder and a prosody
bottleneck to improve prompt representation ability. Second, we propose
Tetra-Diff-Speech, a non-autoregressive structure consisting of four diffusion
model-based modules that design a duration diffusion model to achieve diverse
prosodic expressions. Finally, we propose Tri-Diff-Speech, a non-autoregressive
structure consisting of three diffusion model-based modules that verify the
non-necessity of existing semantic encoding models and achieve the best
results. Experimental results show that our proposed methods outperform
baseline methods. We provide a website with audio samples.
