  Multimodal emotion recognition is a challenging task in emotion computing as
it is quite difficult to extract discriminative features to identify the subtle
differences in human emotions with abstract concept and multiple expressions.
Moreover, how to fully utilize both audio and visual information is still an
open problem. In this paper, we propose a novel multimodal fusion attention
network for audio-visual emotion recognition based on adaptive and multi-level
factorized bilinear pooling (FBP). First, for the audio stream, a fully
convolutional network (FCN) equipped with 1-D attention mechanism and local
response normalization is designed for speech emotion recognition. Next, a
global FBP (G-FBP) approach is presented to perform audio-visual information
fusion by integrating selfattention based video stream with the proposed audio
stream. To improve G-FBP, an adaptive strategy (AG-FBP) to dynamically
calculate the fusion weight of two modalities is devised based on the
emotion-related representation vectors from the attention mechanism of
respective modalities. Finally, to fully utilize the local emotion information,
adaptive and multi-level FBP (AMFBP) is introduced by combining both
global-trunk and intratrunk data in one recording on top of AG-FBP. Tested on
the IEMOCAP corpus for speech emotion recognition with only audio stream, the
new FCN method outperforms the state-ofthe-art results with an accuracy of
71.40%. Moreover, validated on the AFEW database of EmotiW2019 sub-challenge
and the IEMOCAP corpus for audio-visual emotion recognition, the proposed
AM-FBP approach achieves the best accuracy of 63.09% and 75.49% respectively on
the test set.
