  Achieving realistic, vivid, and human-like synthesized conversational
gestures conditioned on multi-modal data is still an unsolved problem due to
the lack of available datasets, models and standard evaluation metrics. To
address this, we build Body-Expression-Audio-Text dataset, BEAT, which has i)
76 hours, high-quality, multi-modal data captured from 30 speakers talking with
eight different emotions and in four different languages, ii) 32 millions
frame-level emotion and semantic relevance annotations. Our statistical
analysis on BEAT demonstrates the correlation of conversational gestures with
facial expressions, emotions, and semantics, in addition to the known
correlation with audio, text, and speaker identity. Based on this observation,
we propose a baseline model, Cascaded Motion Network (CaMN), which consists of
above six modalities modeled in a cascaded architecture for gesture synthesis.
To evaluate the semantic relevancy, we introduce a metric, Semantic Relevance
Gesture Recall (SRGR). Qualitative and quantitative experiments demonstrate
metrics' validness, ground truth data quality, and baseline's state-of-the-art
performance. To the best of our knowledge, BEAT is the largest motion capture
dataset for investigating human gestures, which may contribute to a number of
different research fields, including controllable gesture synthesis,
cross-modality analysis, and emotional gesture recognition. The data, code and
model are available on https://pantomatrix.github.io/BEAT/.
