  Language model (LM) based audio generation frameworks, e.g., AudioLM, have
recently achieved new state-of-the-art performance in zero-shot audio
generation. In this paper, we explore the feasibility of LMs for zero-shot
voice conversion. An intuitive approach is to follow AudioLM - Tokenizing
speech into semantic and acoustic tokens respectively by HuBERT and
SoundStream, and converting source semantic tokens to target acoustic tokens
conditioned on acoustic tokens of the target speaker. However, such an approach
encounters several issues: 1) the linguistic content contained in semantic
tokens may get dispersed during multi-layer modeling while the lengthy speech
input in the voice conversion task makes contextual learning even harder; 2)
the semantic tokens still contain speaker-related information, which may be
leaked to the target speech, lowering the target speaker similarity; 3) the
generation diversity in the sampling of the LM can lead to unexpected outcomes
during inference, leading to unnatural pronunciation and speech quality
degradation. To mitigate these problems, we propose LM-VC, a two-stage language
modeling approach that generates coarse acoustic tokens for recovering the
source linguistic content and target speaker's timbre, and then reconstructs
the fine for acoustic details as converted speech. Specifically, to enhance
content preservation and facilitates better disentanglement, a masked prefix LM
with a mask prediction strategy is used for coarse acoustic modeling. This
model is encouraged to recover the masked content from the surrounding context
and generate target speech based on the target speaker's utterance and
corrupted semantic tokens. Besides, to further alleviate the sampling error in
the generation, an external LM, which employs window attention to capture the
local acoustic relations, is introduced to participate in the coarse acoustic
modeling.
