Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Actionable Conversational Quality Indicators for Improving Task-Oriented
  Dialog Systems   Automatic dialog systems have become a mainstream part of online customer
service. Many such systems are built, maintained, and improved by customer
service specialists, rather than dialog systems engineers and computer
programmers. As conversations between people and machines become commonplace,
it is critical to understand what is working, what is not, and what actions can
be taken to reduce the frequency of inappropriate system responses. These
analyses and recommendations need to be presented in terms that directly
reflect the user experience rather than the internal dialog processing.
  This paper introduces and explains the use of Actionable Conversational
Quality Indicators (ACQIs), which are used both to recognize parts of dialogs
that can be improved, and to recommend how to improve them. This combines
benefits of previous approaches, some of which have focused on producing dialog
quality scoring while others have sought to categorize the types of errors the
dialog system is making.
  We demonstrate the effectiveness of using ACQIs on LivePerson internal dialog
systems used in commercial customer service applications, and on the publicly
available CMU LEGOv2 conversational dataset (Raux et al. 2005). We report on
the annotation and analysis of conversational datasets showing which ACQIs are
important to fix in various situations.
  The annotated datasets are then used to build a predictive model which uses a
turn-based vector embedding of the message texts and achieves an 79% weighted
average f1-measure at the task of finding the correct ACQI for a given
conversation. We predict that if such a model worked perfectly, the range of
potential improvement actions a bot-builder must consider at each turn could be
reduced by an average of 81%.

27
The categories of this paper are: comp-gas
23
The categories of this paper are: solv-int
21
The categories of this paper are: adap-org
36
The categories of this paper are: cmp-lg
11
The categories of this paper are: hep-lat
29
The categories of this paper are: funct-an
37
The categories of this paper are: supr-con
9
The categories of this paper are: nlin
16
The categories of this paper are: stat
34
The categories of this paper are: chem-ph
33
The categories of this paper are: plasm-ph
26
The categories of this paper are: bayes-an
17
The categories of this paper are: q-fin
12
The categories of this paper are: nucl-th
6
The categories of this paper are: astro-ph
35
The categories of this paper are: mtrl-th