
| Predicted       |           |        |           |
|-----------------|-----------|--------|-----------|
| Class           | Occ urred | Recall | Precision |
| TASK SU C CE SS |           |        |           |
| PROBLEMATIC     |           |        |           |

Table Precision and Recall with Exchange Automatic Features salience coverage in predicting problematic dialogues  These features seem to provide good general indicators of the system s success in recognition and understanding The fact that the main focus of the rules is detecting ASR and SLU errors and that none of the DM behaviors are used as predictors also indicates that  in all likelihood  the DM is performing as well as it can  given the noisy input that it is getting from ASR and SLU  An alternative view is that two utterances are not enough to provide meaningful dialogue features such as counts and percentages of reprompts  confirmations  etc One can see that the top two rules use auto-SLU-success  The first rule basically states that if there is no recognition for the second exchange as predicted by the auto-SLU- ss a misunderstanding has been predicted for the second exchange and the system label is DIAL-FOR-ME and the utterance is long then the system will fail  In other words  the stem frequently misinterprets long utterances as DIAL-FOR- ME resulting in task failure gives a subset of the ruleset for the TASK-INDEPT feature set for Exchanges Figure One can see a similarity between this ruleset and the one given in Figure This is due to the fact that when all the automatic features are available  RIPPER has a tendency to pick out the more general task independent ones  with the exception of sys-label  If one compares the second rule in both figures  one can see that RIPPER uses recog-numwords as a substitute for the task specific feature sys-label

## 6.4 Cross-Validation Method Vs. Hand-Labelled-Training Method

As mentioned above an alternative to training the PDP on the automatically derived a uto-
SLU-success feature is to train it on the hand labelled SLU-success while still testing it on the automatic feature This second method is referred to as hand labelled training" and the resulting feature is hlt-SLU-success  This may provide a more accurate model but it may not capture the characteristics of the automatic feature in the test set  Table gives results for the two methods  One can see from this table that there is a slight  insignificant increase in accuracy for Exchange and the whole dialogue using the hand labelled training method However the totally automated method yields a better result cmpared to which as mentioned above  is the most important result for these for Exchanges experiments  This increase shows a trend but is not significant  df The final row of the table gives the results using the hand labelled feature SLU-success in both the training and testing and is taken as the topline result