  Inter-modal interaction plays an indispensable role in multimodal sentiment
analysis. Due to different modalities sequences are usually non-alignment, how
to integrate relevant information of each modality to learn fusion
representations has been one of the central challenges in multimodal learning.
In this paper, a Self-Adjusting Fusion Representation Learning Model (SA-FRLM)
is proposed to learn robust crossmodal fusion representations directly from the
unaligned text and audio sequences. Different from previous works, our model
not only makes full use of the interaction between different modalities but
also maximizes the protection of the unimodal characteristics. Specifically, we
first employ a crossmodal alignment module to project different modalities
features to the same dimension. The crossmodal collaboration attention is then
adopted to model the inter-modal interaction between text and audio sequences
and initialize the fusion representations. After that, as the core unit of the
SA-FRLM, the crossmodal adjustment transformer is proposed to protect original
unimodal characteristics. It can dynamically adapt the fusion representations
by using single modal streams. We evaluate our approach on the public
multimodal sentiment analysis datasets CMU-MOSI and CMU-MOSEI. The experiment
results show that our model has significantly improved the performance of all
the metrics on the unaligned text-audio sequences.
