  Conversational agents are exploding in popularity. However, much work remains
in the area of non goal-oriented conversations, despite significant growth in
research interest over recent years. To advance the state of the art in
conversational AI, Amazon launched the Alexa Prize, a 2.5-million dollar
university competition where sixteen selected university teams built
conversational agents to deliver the best social conversational experience.
Alexa Prize provided the academic community with the unique opportunity to
perform research with a live system used by millions of users. The subjectivity
associated with evaluating conversations is key element underlying the
challenge of building non-goal oriented dialogue systems. In this paper, we
propose a comprehensive evaluation strategy with multiple metrics designed to
reduce subjectivity by selecting metrics which correlate well with human
judgement. The proposed metrics provide granular analysis of the conversational
agents, which is not captured in human ratings. We show that these metrics can
be used as a reasonable proxy for human judgment. We provide a mechanism to
unify the metrics for selecting the top performing agents, which has also been
applied throughout the Alexa Prize competition. To our knowledge, to date it is
the largest setting for evaluating agents with millions of conversations and
hundreds of thousands of ratings from users. We believe that this work is a
step towards an automatic evaluation process for conversational AIs.
