  This paper proposes a novel direct Audio-Visual Speech to Audio-Visual Speech
Translation (AV2AV) framework, where the input and output of the system are
multimodal (i.e., audio and visual speech). With the proposed AV2AV, two key
advantages can be brought: 1) We can perform real-like conversations with
individuals worldwide in a virtual meeting by utilizing our own primary
languages. In contrast to Speech-to-Speech Translation (A2A), which solely
translates between audio modalities, the proposed AV2AV directly translates
between audio-visual speech. This capability enhances the dialogue experience
by presenting synchronized lip movements along with the translated speech. 2)
We can improve the robustness of the spoken language translation system. By
employing the complementary information of audio-visual speech, the system can
effectively translate spoken language even in the presence of acoustic noise,
showcasing robust performance. To mitigate the problem of the absence of a
parallel AV2AV translation dataset, we propose to train our spoken language
translation system with the audio-only dataset of A2A. This is done by learning
unified audio-visual speech representations through self-supervised learning in
advance to train the translation system. Moreover, we propose an AV-Renderer
that can generate raw audio and video in parallel. It is designed with
zero-shot speaker modeling, thus the speaker in source audio-visual speech can
be maintained at the target translated audio-visual speech. The effectiveness
of AV2AV is evaluated with extensive experiments in a many-to-many language
translation setting. The demo page is available on
https://choijeongsoo.github.io/av2av.
