  The virtual world is being established in which digital humans are created
indistinguishable from real humans. Producing their audio-related capabilities
is crucial since voice conveys extensive personal characteristics. We aim to
create a controllable audio-form virtual singer; however, supervised modeling
and controlling all different factors of the singing voice, such as timbre,
tempo, pitch, and lyrics, is extremely difficult since accurately labeling all
such information needs enormous labor work. In this paper, we propose a
framework that could digitize a person's voice by simply "listening" to the
clean voice recordings of any content in a fully unsupervised manner and
predict singing voices even only using speaking recordings. A variational
auto-encoder (VAE) based framework is developed, which leverages a set of
pre-trained models to encode the audio as various hidden embeddings
representing different factors of the singing voice, and further decodes the
embeddings into raw audio. By manipulating the hidden embeddings for different
factors, the resulting singing voices can be controlled, and new virtual
singers can also be further generated by interpolating between timbres.
Evaluations of different types of experiments demonstrate the proposed method's
effectiveness. The proposed method is the critical technique for producing the
AI choir, which empowered the human-AI symbiotic orchestra in Hong Kong in July
2022.
