  In recent years, deep learning models have been applied to neuroimaging data
for early diagnosis of Alzheimer's disease (AD). Structural magnetic resonance
imaging (sMRI) and positron emission tomography (PET) images provide structural
and functional information about the brain, respectively. Combining these
features leads to improved performance than using a single modality alone in
building predictive models for AD diagnosis. However, current multi-modal
approaches in deep learning, based on sMRI and PET, are mostly limited to
convolutional neural networks, which do not facilitate integration of both
image and phenotypic information of subjects. We propose to use graph neural
networks (GNN) that are designed to deal with problems in non-Euclidean
domains. In this study, we demonstrate how brain networks can be created from
sMRI or PET images and be used in a population graph framework that can combine
phenotypic information with imaging features of these brain networks. Then, we
present a multi-modal GNN framework where each modality has its own branch of
GNN and a technique is proposed to combine the multi-modal data at both the
level of node vectors and adjacency matrices. Finally, we perform late fusion
to combine the preliminary decisions made in each branch and produce a final
prediction. As multi-modality data becomes available, multi-source and
multi-modal is the trend of AD diagnosis. We conducted explorative experiments
based on multi-modal imaging data combined with non-imaging phenotypic
information for AD diagnosis and analyzed the impact of phenotypic information
on diagnostic performance. Results from experiments demonstrated that our
proposed multi-modal approach improves performance for AD diagnosis, and this
study also provides technical reference and support the need for multivariate
multi-modal diagnosis methods.
