  Many real-world problems are inherently multimodal, from spoken language,
gestures, and paralinguistics humans use to communicate, to force,
proprioception, and visual sensors on robots. While there has been an explosion
of interest in multimodal learning, these methods are focused on a small set of
modalities primarily in language, vision, and audio. In order to accelerate
generalization towards diverse and understudied modalities, this paper studies
efficient representation learning for high-modality scenarios involving a large
set of diverse modalities. Since adding new models for every new modality
becomes prohibitively expensive, a critical technical challenge is
heterogeneity quantification: how can we measure which modalities encode
similar information and interactions in order to permit parameter sharing with
previous modalities? This paper proposes two new information theoretic metrics
for heterogeneity quantification: (1) modality heterogeneity studies how
similar 2 modalities {X1,X2} are by measuring how much information can be
transferred from X1 to X2, while (2) interaction heterogeneity studies how
similarly pairs of modalities {X1,X2}, {X3,X4} interact by measuring how much
information can be transferred from fusing {X1,X2} to {X3,X4}. We show the
importance of these 2 proposed metrics as a way to automatically prioritize the
fusion of modalities that contain unique information or interactions. The
result is a single model, HighMMT, that scales up to 10 modalities (text,
image, audio, video, sensors, proprioception, speech, time-series, sets, and
tables) and 15 tasks from 5 research areas. Not only does HighMMT outperform
prior methods on the tradeoff between performance and efficiency, it also
demonstrates a crucial scaling behavior: performance continues to improve with
each modality added, and it transfers to entirely new modalities and tasks
during fine-tuning.
