  Speech emotion recognition~(SER) refers to the technique of inferring the
emotional state of an individual from speech signals. SERs continue to garner
interest due to their wide applicability. Although the domain is mainly founded
on signal processing, machine learning, and deep learning, generalizing over
languages continues to remain a challenge. However, developing generalizable
and transferable models are critical due to a lack of sufficient resources in
terms of data and labels for languages beyond the most commonly spoken ones. To
improve performance over languages, we propose a denoising autoencoder with
semi-supervision using a continuous metric loss based on either activation or
valence. The novelty of this work lies in our proposal of continuous metric
learning, which is among the first proposals on the topic to the best of our
knowledge. Furthermore, to address the lack of activation and valence labels in
the transfer datasets, we annotate the signal samples with activation and
valence levels corresponding to a dimensional model of emotions, which were
then used to evaluate the quality of the embedding over the transfer datasets.
We show that the proposed semi-supervised model consistently outperforms the
baseline unsupervised method, which is a conventional denoising autoencoder, in
terms of emotion classification accuracy as well as correlation with respect to
the dimensional variables. Further evaluation of classification accuracy with
respect to the reference, a BERT based speech representation model, shows that
the proposed method is comparable to the reference method in classifying
specific emotion classes at a much lower complexity.
