  Audio-Visual Emotion Recognition (AVER) has garnered increasing attention in
recent years for its critical role in creating emotion-ware intelligent
machines. Previous efforts in this area are dominated by the supervised
learning paradigm. Despite significant progress, supervised learning is meeting
its bottleneck due to the longstanding data scarcity issue in AVER. Motivated
by recent advances in self-supervised learning, we propose Hierarchical
Contrastive Masked Autoencoder (HiCMAE), a novel self-supervised framework that
leverages large-scale self-supervised pre-training on vast unlabeled
audio-visual data to promote the advancement of AVER. Following prior arts in
self-supervised audio-visual representation learning, HiCMAE adopts two primary
forms of self-supervision for pre-training, namely masked data modeling and
contrastive learning. Unlike them which focus exclusively on top-layer
representations while neglecting explicit guidance of intermediate layers,
HiCMAE develops a three-pronged strategy to foster hierarchical audio-visual
feature learning and improve the overall quality of learned representations. To
verify the effectiveness of HiCMAE, we conduct extensive experiments on 9
datasets covering both categorical and dimensional AVER tasks. Experimental
results show that our method significantly outperforms state-of-the-art
supervised and self-supervised audio-visual methods, which indicates that
HiCMAE is a powerful audio-visual emotion representation learner. Codes and
models will be publicly available at https://github.com/sunlicai/HiCMAE.
