  Investigating the relationship between internal tissue point motion of the
tongue and oropharyngeal muscle deformation measured from tagged MRI and
intelligible speech can aid in advancing speech motor control theories and
developing novel treatment methods for speech related-disorders. However,
elucidating the relationship between these two sources of information is
challenging, due in part to the disparity in data structure between
spatiotemporal motion fields (i.e., 4D motion fields) and one-dimensional audio
waveforms. In this work, we present an efficient encoder-decoder translation
network for exploring the predictive information inherent in 4D motion fields
via 2D spectrograms as a surrogate of the audio data. Specifically, our encoder
is based on 3D convolutional spatial modeling and transformer-based temporal
modeling. The extracted features are processed by an asymmetric 2D convolution
decoder to generate spectrograms that correspond to 4D motion fields.
Furthermore, we incorporate a generative adversarial training approach into our
framework to further improve synthesis quality on our generated spectrograms.
We experiment on 63 paired motion field sequences and speech waveforms,
demonstrating that our framework enables the generation of clear audio
waveforms from a sequence of motion fields. Thus, our framework has the
potential to improve our understanding of the relationship between these two
modalities and inform the development of treatments for speech disorders.
