label age age and gender gender of the user the actual modality of the user utterance user-modality  one of  nothing  speech  touchtone  speech   touchtone  non speech   and a cleaned transcript with non word noise information removed  clean-tscript  From these fea tures  we calculated two derived features  The first was the number of words in the cleaned transcript  cltscript-numwords   again on the assumption that utterance length is strongly correlated with ASR and SLU errors The second derived feature was based on calculating whether the human-label matches the sys-label from the Dialogue Manager  SLU-success This feature is described in detail in the next section In the experiments the features in Figure - excluding the hand labelled features are referred to as the AUTOMATIC feature set The experiments test how well misunderstandings can be identified and whether problematic dialogues can be predicted using the AUTOMATIC
features We compare the performance of the AUTOMATIC feature set to the full feature set including the hand labelled features and to the performance of the AUTOMATIC feature set with and without the auto-SLU-success feature Figure   gives an example of the encoding of some of the automatic features for the second exchange of the WIZARD dialogue in Figure The prefix e2-" designates the second exchange We discuss several of the features values here to ensure that the reader understands the way in which the features are used  In utterance S  in Figure    the system says Sorry please briefly tell me how I may help you this is encoded by several features  The feature e2-prompt gives the name of In Figure that prompt top-reject-rep The feature e2-reprompt specifies that S is a reprompt a second attempt by the system to elicit a description of the caller s problem The feature e2-confirm specifies that S  is not a confirmation prompt  The feature e2-subdial specifies that S initiates a subdialogue and e2-num-subdials encodes that this is the first subdialogue ssss of them initiate subdialogues As mentioned earlier we are also interested in generalizing our problematic dialogue predictor to other systems  Thus  we trained RIPPER using only features that are both automatically acquirable during runtime and independent of the HMIHY task The subset of features from Figure   that fit this qualification are in Figure  
We refer to them as the AUTO, TASK-INDEPT feature set Examples of features that are not task independent include recog-grammar, sys-label, prompt and the hand labelled features

## 5.  Auto-Slu-Success Predictor

The goal of the auto-SLU-success predictor is to identify for each exchange  whether or not the system correctly understood the users utterance As mentioned above  when the dialogues were transcribed by humans after the data collection was completed  the human labelers not only transcribed the users utterances  but also labelled each utterance with a semantic category representing the task that the user was asking HMIHY to perform This label is called the human-label The system's Dialogue Manager decides among several different hypotheses produced by the SLU module  and logs its hypothesis about what task the user was asking HMIHY to perform the Dialogue Manager s hypothesis is known as the sys-label We distinguish four classes of spoken language understanding outcomes based on comparing the human-label the sys-label and recognition results for card and telephone RCORRECT  SLU correctly identified the task and any digit strings were also nm