  Estimating dimensional emotions, such as activation, valence and dominance,
from acoustic speech signals has been widely explored over the past few years.
While accurate estimation of activation and dominance from speech seem to be
possible, the same for valence remains challenging. Previous research has shown
that the use of lexical information can improve valence estimation performance.
Lexical information can be obtained from pre-trained acoustic models, where the
learned representations can improve valence estimation from speech. We
investigate the use of pre-trained model representations to improve valence
estimation from acoustic speech signal. We also explore fusion of
representations to improve emotion estimation across all three emotion
dimensions: activation, valence and dominance. Additionally, we investigate if
representations from pre-trained models can be distilled into models trained
with low-level features, resulting in models with a less number of parameters.
We show that fusion of pre-trained model embeddings result in a 79% relative
improvement in concordance correlation coefficient CCC on valence estimation
compared to standard acoustic feature baseline (mel-filterbank energies), while
distillation from pre-trained model embeddings to lower-dimensional
representations yielded a relative 12% improvement. Such performance gains were
observed over two evaluation sets, indicating that our proposed architecture
generalizes across those evaluation sets. We report new state-of-the-art
"text-free" acoustic-only dimensional emotion estimation $CCC$ values on two
MSP-Podcast evaluation sets.
