  We propose SelfVC, a training strategy to iteratively improve a voice
conversion model with self-synthesized examples. Previous efforts on voice
conversion focus on explicitly disentangling speech representations to
separately encode speaker characteristics and linguistic content. However,
disentangling speech representations to capture such attributes using
task-specific loss terms can lead to information loss by discarding finer
nuances of the original signal. In this work, instead of explicitly
disentangling attributes with loss terms, we present a framework to train a
controllable voice conversion model on entangled speech representations derived
from self-supervised learning and speaker verification models. First, we
develop techniques to derive prosodic information from the audio signal and SSL
representations to train predictive submodules in the synthesis model. Next, we
propose a training strategy to iteratively improve the synthesis model for
voice conversion, by creating a challenging training objective using
self-synthesized examples. In this training approach, the current state of the
synthesis model is used to generate voice-converted variations of an utterance,
which serve as inputs for the reconstruction task, ensuring a continuous and
purposeful refinement of the model. We demonstrate that incorporating such
self-synthesized examples during training improves the speaker similarity of
generated speech as compared to a baseline voice conversion model trained
solely on heuristically perturbed inputs. SelfVC is trained without any text
and is applicable to a range of tasks such as zero-shot voice conversion,
cross-lingual voice conversion, and controllable speech synthesis with pitch
and pace modifications. SelfVC achieves state-of-the-art results in zero-shot
voice conversion on metrics evaluating naturalness, speaker similarity, and
intelligibility of synthesized audio.
