  Multi-modal Contrastive Representation learning aims to encode different
modalities into a semantically aligned shared space. This paradigm shows
remarkable generalization ability on numerous downstream tasks across various
modalities. However, the reliance on massive high-quality data pairs limits its
further development on more modalities. This paper proposes a novel
training-efficient method for learning MCR without paired data called
Connecting Multi-modal Contrastive Representations (C-MCR). Specifically, given
two existing MCRs pre-trained on (A, B) and (B, C) modality pairs, we project
them to a new space and use the data from the overlapping modality B to
aligning the two MCRs in the new space. Meanwhile, since the modality pairs (A,
B) and (B, C) are already aligned within each MCR, the connection learned by
overlapping modality can also be transferred to non-overlapping modality pair
(A, C). To unleash the potential of C-MCR, we further introduce a
semantic-enhanced inter- and intra-MCR connection method. We first enhance the
semantic consistency and completion of embeddings across different modalities
for more robust alignment. Then we utilize the inter-MCR alignment to establish
the connection, and employ the intra-MCR alignment to better maintain the
connection for inputs from non-overlapping modalities. To demonstrate the
effectiveness of C-MCR, we connect CLIP and CLAP via texts to derive
audio-visual representations, and integrate CLIP and ULIP via images for
3D-language representations. Remarkably, without using any paired data, C-MCR
for audio-visual achieves state-of-the-art performance on audio-image
retrieval, audio-visual source localization, and counterfactual audio-image
recognition tasks. Furthermore, C-MCR for 3D-language also attains advanced
zero-shot 3D point cloud classification accuracy on ModelNet40.
