  Visually grounded speech systems learn from paired images and their spoken
captions. Recently, there have been attempts to utilize the visually grounded
models trained from images and their corresponding text captions, such as CLIP,
to improve speech-based visually grounded models' performance. However, the
majority of these models only utilize the pretrained image encoder. Cascaded
SpeechCLIP attempted to generate localized word-level information and utilize
both the pretrained image and text encoders. Despite using both, they noticed a
substantial drop in retrieval performance. We proposed Segmental SpeechCLIP
which used a hierarchical segmental speech encoder to generate sequences of
word-like units. We used the pretrained CLIP text encoder on top of these
word-like unit representations and showed significant improvements over the
cascaded variant of SpeechCLIP. Segmental SpeechCLIP directly learns the word
embeddings as input to the CLIP text encoder bypassing the vocabulary
embeddings. Here, we explore mapping audio to CLIP vocabulary embeddings via
regularization and quantization. As our objective is to distill semantic
information into the speech encoders, we explore the usage of large unimodal
pretrained language models as the text encoders. Our method enables us to
bridge image and text encoders e.g. DINO and RoBERTa trained with uni-modal
data. Finally, we extend our framework in audio-only settings where only pairs
of semantically related audio are available. Experiments show that audio-only
systems perform close to the audio-visual system.
