  In recent years, the rapid growth of online multimedia services, such as
e-commerce platforms, has necessitated the development of personalised
recommendation approaches that can encode diverse content about each item.
Indeed, modern multi-modal recommender systems exploit diverse features
obtained from raw images and item descriptions to enhance the recommendation
performance. However, the existing multi-modal recommenders primarily depend on
the features extracted individually from different media through pre-trained
modality-specific encoders, and exhibit only shallow alignments between
different modalities - limiting these systems' ability to capture the
underlying relationships between the modalities. In this paper, we investigate
the usage of large multi-modal encoders within the specific context of
recommender systems, as these have previously demonstrated state-of-the-art
effectiveness when ranking items across various domains. Specifically, we
tailor two state-of-the-art multi-modal encoders (CLIP and VLMo) for
recommendation tasks using a range of strategies, including the exploration of
pre-trained and fine-tuned encoders, as well as the assessment of the
end-to-end training of these encoders. We demonstrate that pre-trained large
multi-modal encoders can generate more aligned and effective user/item
representations compared to existing modality-specific encoders across three
multi-modal recommendation datasets. Furthermore, we show that fine-tuning
these large multi-modal encoders with recommendation datasets leads to an
enhanced recommendation performance. In terms of different training paradigms,
our experiments highlight the essential role of the end-to-end training of
large multi-modal encoders in multi-modal recommendation systems.
