
## 7. Related Work

The research reported here is the first that we know of to automatically analyze a corpus of logs from a spoken dialogue system for the purpose of learning to predict problematic situations This work builds on two strands of earlier research First this approach was inspired by work on the PARADISE evaluation framework for spoken dialogue systems which utilizes both multivariate linear regression and CART to predict user satisfaction as a ofunction of a number of other metrics  Walker  Litman  Kamm Abella Walker a Research using PARADISE has found that task completion is always a major et al predictor of user satisfaction  and has examined predictors of task completion  Here  our goals are similar in that we attempt to understand the factors that predict task completion Secondly this work builds on earlier research on learning to identify dialogues in which the user experienced poor speech recognizer performance  Litman et al Because that work was based on features synthesized over the entire dialogue  the hypotheses that were learned could not be used for prediction during runtime In addition in contrast to the current study the previous work automatically approximated the notion of a good or bad dialogue using a threshold on the percentage of recognition errors  There is a danger of this approach being circular when recognition performance at the utterance level is a primary predictor of a good or bad dialogue In this work the notion of a good TASKSUCCESS and bad  problematic  dialogue was labelled by humans In previous work  Walker et al b reported results from training a problematic dialogue predictor in which they noted the extent to which the hand labelled SLU-success feature improves classifier performance As a result of this prior analysis in this work we report results from training an auto-SLU-success classifier for each exchange and using its predictions as an input feature to the Problematic Dialogue Predictor There are a number of previous studies on predicting recognition errors and user corrections whi are related to the auto-SL U-success predictor that we report on here  Hirschberg et al Hirschberg Litman Swerts b Levow Litman Hirschberg Swerts Swerts Hirschberg Litman Hirschberg et al apply RIPPER to predict recognition errors in a corpus of utterances In contrast to our work they utilize prosodic features in combination with acoustic confidence scores They report a best classifier accuracy of which is a ing in the This result can be compared with our binary auto¬
SLU-success predictor  RCORRECT vs  RINCORRECT  discussed in Section     Examination of the rules learned by their classifier suggests that durational features are important   While we do not use amplitude or F features we do have an asr-duration feature which is logged by the recognizer Without any of the other prosodic features the auto-SLU-success predictor has an accuracy of improvement over the baseline of It is possible a that including prosodic features in the auto-SLU-success predictor could improve this result en further Previous studies on error correction recognition are also related to our method of mis understanding recognition Levow applied similar techniques to learn to distinguish between utterances in which the user originally provided some information to the system and corrections  which provided the same information a second time  following a misunder standing This may be more related to our research than it first appears since corrections