  Multimodal emotion recognition has recently gained much attention since it
can leverage diverse and complementary relationships over multiple modalities
(e.g., audio, visual, biosignals, etc.), and can provide some robustness to
noisy modalities. Most state-of-the-art methods for audio-visual (A-V) fusion
rely on recurrent networks or conventional attention mechanisms that do not
effectively leverage the complementary nature of A-V modalities. In this paper,
we focus on dimensional emotion recognition based on the fusion of facial and
vocal modalities extracted from videos. Specifically, we propose a joint
cross-attention model that relies on the complementary relationships to extract
the salient features across A-V modalities, allowing for accurate prediction of
continuous values of valence and arousal. The proposed fusion model efficiently
leverages the inter-modal relationships, while reducing the heterogeneity
between the features. In particular, it computes the cross-attention weights
based on correlation between the combined feature representation and individual
modalities. By deploying the combined A-V feature representation into the
cross-attention module, the performance of our fusion module improves
significantly over the vanilla cross-attention module. Experimental results on
validation-set videos from the AffWild2 dataset indicate that our proposed A-V
fusion model provides a cost-effective solution that can outperform
state-of-the-art approaches. The code is available on GitHub:
https://github.com/praveena2j/JointCrossAttentional-AV-Fusion.
