  Brain network analysis is a useful approach to studying human brain disorders
because it can distinguish patients from healthy people by detecting abnormal
connections. Due to the complementary information from multiple modal
neuroimages, multimodal fusion technology has a lot of potential for improving
prediction performance. However, effective fusion of multimodal medical images
to achieve complementarity is still a challenging problem. In this paper, a
novel hierarchical structural-functional connectivity fusing (HSCF) model is
proposed to construct brain structural-functional connectivity matrices and
predict abnormal brain connections based on functional magnetic resonance
imaging (fMRI) and diffusion tensor imaging (DTI). Specifically, the prior
knowledge is incorporated into the separators for disentangling each modality
of information by the graph convolutional networks (GCN). And a disentangled
cosine distance loss is devised to ensure the disentanglement's effectiveness.
Moreover, the hierarchical representation fusion module is designed to
effectively maximize the combination of relevant and effective features between
modalities, which makes the generated structural-functional connectivity more
robust and discriminative in the cognitive disease analysis. Results from a
wide range of tests performed on the public Alzheimer's Disease Neuroimaging
Initiative (ADNI) database show that the proposed model performs better than
competing approaches in terms of classification evaluation. In general, the
proposed HSCF model is a promising model for generating brain
structural-functional connectivities and identifying abnormal brain connections
as cognitive disease progresses.
