  Automated emotion detection in speech is a challenging task due to the
complex interdependence between words and the manner in which they are spoken.
It is made more difficult by the available datasets; their small size and
incompatible labeling idiosyncrasies make it hard to build generalizable
emotion detection systems. To address these two challenges, we present a
multi-modal approach that first transfers learning from related tasks in speech
and text to produce robust neural embeddings and then uses these embeddings to
train a pLDA classifier that is able to adapt to previously unseen emotions and
domains. We begin by training a multilayer TDNN on the task of speaker
identification with the VoxCeleb corpora and then fine-tune it on the task of
emotion identification with the Crema-D corpus. Using this network, we extract
speech embeddings for Crema-D from each of its layers, generate and concatenate
text embeddings for the accompanying transcripts using a fine-tuned BERT model
and then train an LDA - pLDA classifier on the resulting dense representations.
We exhaustively evaluate the predictive power of every component: the TDNN
alone, speech embeddings from each of its layers alone, text embeddings alone
and every combination thereof. Our best variant, trained on only VoxCeleb and
Crema-D and evaluated on IEMOCAP, achieves an EER of 38.05%. Including a
portion of IEMOCAP during training produces a 5-fold averaged EER of 25.72%
(For comparison, 44.71% of the gold-label annotations include at least one
annotator who disagrees).
