  This paper reports on the GENEA Challenge 2023, in which participating teams
built speech-driven gesture-generation systems using the same speech and motion
dataset, followed by a joint evaluation. This year's challenge provided data on
both sides of a dyadic interaction, allowing teams to generate full-body motion
for an agent given its speech (text and audio) and the speech and motion of the
interlocutor. We evaluated 12 submissions and 2 baselines together with
held-out motion-capture data in several large-scale user studies. The studies
focused on three aspects: 1) the human-likeness of the motion, 2) the
appropriateness of the motion for the agent's own speech whilst controlling for
the human-likeness of the motion, and 3) the appropriateness of the motion for
the behaviour of the interlocutor in the interaction, using a setup that
controls for both the human-likeness of the motion and the agent's own speech.
We found a large span in human-likeness between challenge submissions, with a
few systems rated close to human mocap. Appropriateness seems far from being
solved, with most submissions performing in a narrow range slightly above
chance, far behind natural motion. The effect of the interlocutor is even more
subtle, with submitted systems at best performing barely above chance.
Interestingly, a dyadic system being highly appropriate for agent speech does
not necessarily imply high appropriateness for the interlocutor. Additional
material is available via the project website at
https://svito-zar.github.io/GENEAchallenge2023/ .
