  We propose an end-to-end affect recognition approach using a Convolutional
Neural Network (CNN) that handles multiple languages, with applications to
emotion and personality recognition from speech. We lay the foundation of a
universal model that is trained on multiple languages at once. As affect is
shared across all languages, we are able to leverage shared information between
languages and improve the overall performance for each one. We obtained an
average improvement of 12.8% on emotion and 10.1% on personality when compared
with the same model trained on each language only. It is end-to-end because we
directly take narrow-band raw waveforms as input. This allows us to accept as
input audio recorded from any source and to avoid the overhead and information
loss of feature extraction. It outperforms a similar CNN using spectrograms as
input by 12.8% for emotion and 6.3% for personality, based on F-scores.
Analysis of the network parameters and layers activation shows that the network
learns and extracts significant features in the first layer, in particular
pitch, energy and contour variations. Subsequent convolutional layers instead
capture language-specific representations through the analysis of
supra-segmental features. Our model represents an important step for the
development of a fully universal affect recognizer, able to recognize
additional descriptors, such as stress, and for the future implementation into
affective interactive systems.
