  Disentanglement of a speaker's timbre and style is very important for style
transfer in multi-speaker multi-style text-to-speech (TTS) scenarios. With the
disentanglement of timbres and styles, TTS systems could synthesize expressive
speech for a given speaker with any style which has been seen in the training
corpus. However, there are still some shortcomings with the current research on
timbre and style disentanglement. The current method either requires
single-speaker multi-style recordings, which are difficult and expensive to
collect, or uses a complex network and complicated training method, which is
difficult to reproduce and control the style transfer behavior. To improve the
disentanglement effectiveness of timbres and styles, and to remove the reliance
on single-speaker multi-style corpus, a simple but effective timbre and style
disentanglement method is proposed in this paper. The FastSpeech2 network is
employed as the backbone network, with explicit duration, pitch, and energy
trajectory to represent the style. Each speaker's data is considered as a
separate and isolated style, then a speaker embedding and a style embedding are
added to the FastSpeech2 network to learn disentangled representations.
Utterance level pitch and energy normalization are utilized to improve the
decoupling effect. Experimental results demonstrate that the proposed model
could synthesize speech with any style seen during training with high style
similarity while maintaining very high speaker similarity.
