  Direct speech-to-speech translation (S2ST) translates speech from one
language into another using a single model. However, due to the presence of
linguistic and acoustic diversity, the target speech follows a complex
multimodal distribution, posing challenges to achieving both high-quality
translations and fast decoding speeds for S2ST models. In this paper, we
propose DASpeech, a non-autoregressive direct S2ST model which realizes both
fast and high-quality S2ST. To better capture the complex distribution of the
target speech, DASpeech adopts the two-pass architecture to decompose the
generation process into two steps, where a linguistic decoder first generates
the target text, and an acoustic decoder then generates the target speech based
on the hidden states of the linguistic decoder. Specifically, we use the
decoder of DA-Transformer as the linguistic decoder, and use FastSpeech 2 as
the acoustic decoder. DA-Transformer models translations with a directed
acyclic graph (DAG). To consider all potential paths in the DAG during
training, we calculate the expected hidden states for each target token via
dynamic programming, and feed them into the acoustic decoder to predict the
target mel-spectrogram. During inference, we select the most probable path and
take hidden states on that path as input to the acoustic decoder. Experiments
on the CVSS Fr-En benchmark demonstrate that DASpeech can achieve comparable or
even better performance than the state-of-the-art S2ST model Translatotron 2,
while preserving up to 18.53x speedup compared to the autoregressive baseline.
Compared with the previous non-autoregressive S2ST model, DASpeech does not
rely on knowledge distillation and iterative decoding, achieving significant
improvements in both translation quality and decoding speed. Furthermore,
DASpeech shows the ability to preserve the speaker's voice of the source speech
during translation.
