  This paper proposes a novel multimodal self-supervised architecture for
energy-efficient audio-visual (AV) speech enhancement that integrates Graph
Neural Networks with canonical correlation analysis (CCA-GNN). The proposed
approach lays its foundations on a state-of-the-art CCA-GNN that learns
representative embeddings by maximizing the correlation between pairs of
augmented views of the same input while decorrelating disconnected features.
The key idea of the conventional CCA-GNN involves discarding
augmentation-variant information and preserving augmentation-invariant
information while preventing capturing of redundant information. Our proposed
AV CCA-GNN model deals with multimodal representation learning context.
Specifically, our model improves contextual AV speech processing by maximizing
canonical correlation from augmented views of the same channel and canonical
correlation from audio and visual embeddings. In addition, it proposes a
positional node encoding that considers a prior-frame sequence distance instead
of a feature-space representation when computing the node's nearest neighbors,
introducing temporal information in the embeddings through the neighborhood's
connectivity. Experiments conducted on the benchmark ChiME3 dataset show that
our proposed prior frame-based AV CCA-GNN ensures better feature learning in
the temporal context, leading to more energy-efficient speech reconstruction
than state-of-the-art CCA-GNN and multilayer perceptron.
