  Recent neuroimaging studies that focus on predicting brain disorders via
modern machine learning approaches commonly include a single modality and rely
on supervised over-parameterized models.However, a single modality provides
only a limited view of the highly complex brain. Critically, supervised models
in clinical settings lack accurate diagnostic labels for training. Coarse
labels do not capture the long-tailed spectrum of brain disorder phenotypes,
which leads to a loss of generalizability of the model that makes them less
useful in diagnostic settings. This work presents a novel multi-scale
coordinated framework for learning multiple representations from multimodal
neuroimaging data. We propose a general taxonomy of informative inductive
biases to capture unique and joint information in multimodal self-supervised
fusion. The taxonomy forms a family of decoder-free models with reduced
computational complexity and a propensity to capture multi-scale relationships
between local and global representations of the multimodal inputs. We conduct a
comprehensive evaluation of the taxonomy using functional and structural
magnetic resonance imaging (MRI) data across a spectrum of Alzheimer's disease
phenotypes and show that self-supervised models reveal disorder-relevant brain
regions and multimodal links without access to the labels during pre-training.
The proposed multimodal self-supervised learning yields representations with
improved classification performance for both modalities. The concomitant rich
and flexible unsupervised deep learning framework captures complex multimodal
relationships and provides predictive performance that meets or exceeds that of
a more narrow supervised classification analysis. We present elaborate
quantitative evidence of how this framework can significantly advance our
search for missing links in complex brain disorders.
