  Leveraging the fact that speaker identity and content vary on different time
scales, \acrlong{fhvae} (\acrshort{fhvae}) uses different latent variables to
symbolize these two attributes. Disentanglement of these attributes is carried
out by different prior settings of the corresponding latent variables. For the
prior of speaker identity variable, \acrshort{fhvae} assumes it is a Gaussian
distribution with an utterance-scale varying mean and a fixed variance. By
setting a small fixed variance, the training process promotes identity
variables within one utterance gathering close to the mean of their prior.
However, this constraint is relatively weak, as the mean of the prior changes
between utterances. Therefore, we introduce contrastive learning into the
\acrshort{fhvae} framework, to make the speaker identity variables gathering
when representing the same speaker, while distancing themselves as far as
possible from those of other speakers. The model structure has not been changed
in this work but only the training process, thus no additional cost is needed
during testing. Voice conversion has been chosen as the application in this
paper. Latent variable evaluations include speaker verification and
identification for the speaker identity variable, and speech recognition for
the content variable. Furthermore, assessments of voice conversion performance
are on the grounds of fake speech detection experiments. Results show that the
proposed method improves both speaker identity and content feature extraction
compared to \acrshort{fhvae}, and has better performance than baseline on
conversion.
