  Speech classification tasks often require powerful language understanding
models to grasp useful features, which becomes problematic when limited
training data is available. To attain superior classification performance, we
propose to harness the inherent value of multimodal representations by
transcribing speech using automatic speech recognition (ASR) models and
translating the transcripts into different languages via pretrained translation
models. We thus obtain an audio-textual (multimodal) representation for each
data sample. Subsequently, we combine language-specific Bidirectional Encoder
Representations from Transformers (BERT) with Wav2Vec2.0 audio features via a
novel cascaded cross-modal transformer (CCMT). Our model is based on two
cascaded transformer blocks. The first one combines text-specific features from
distinct languages, while the second one combines acoustic features with
multilingual features previously learned by the first transformer block. We
employed our system in the Requests Sub-Challenge of the ACM Multimedia 2023
Computational Paralinguistics Challenge. CCMT was declared the winning
solution, obtaining an unweighted average recall (UAR) of 65.41% and 85.87% for
complaint and request detection, respectively. Moreover, we applied our
framework on the Speech Commands v2 and HarperValleyBank dialog data sets,
surpassing previous studies reporting results on these benchmarks. Our code is
freely available for download at: https://github.com/ristea/ccmt.
