spoke   The RCORRECT class accounts for of the exchanges in the corpus  The of the exchanges   The RMISMATCH class accounts RPARTIAL- MATCH accounts for of the exchanges and the no-recog class accounts for for of the exchanges The auto-SLU-success predictor is trained using    fully automatic features These features are the Acoustic ASR features SLU features and Dialogue Manager and Discourse History features  given in Figure      Hand labelled features were not used We evaluate the four way auto-SLU-success classifier by reporting accuracy precision recall and the categorization confusion matrix This classifier is trained on all the features for the whole training set and then tested on the held out test set Table summarizes the overall accuracy results of the system trained on the whole to training set and tested on the test set described in Section The first line of Table represents the accuracy from always guessing the majority class  NO-RECOG   this is the BASELINE against which the other results should be compared  The second row  labelled A A T O MATIC shows the accuracy based on using all the features available from the system modules  This classifier can identify SLU errors better than the baseline  An exper iment was run to see if the cross validation method described in Section performs worse than using the whole data on the same test set This experiment showed that there was little loss of accuracy when using cross validation

| Features  Used           | Accuracy   |
|--------------------------|------------|
| BASELINE  majority class |            |
| AU TO MATIC              |            |

![0_image_0.png](0_image_0.png)

![0_image_1.png](0_image_1.png)

## Table

Results for detecting SLU Errors using RIPPER
shows some top performing rules that RIPPER learns when given all the fea Figure tures  These rules directly reflect the usefulness of the SLU features  Note that some of the rules use ASR features in combination with SLU features such as salpertime Previous studies c have also shown SLU features to be useful  We had also hypothesized Walker et al that features from the Dialogue Manager and the discourse history might be useful predic tors of SLU errors  however these features rarely appear in the rules with the exception of sys-label  This is in accordance with previous experiments which show that these features do not add significantly to the performance of the SLU ONLY feature set  Walker et al We also report precision and recall for each category on the held out test set  The results are shown in Tables and Table shows that the classification accuracy rate is a result of a high rate of correct classification for the RCORRECT and NO-RECOG class  at the cost of a lower rate for RMISMATCH and RPARTIAL-MATCH This is probably due to the fact that there are fewer examples of these categories in the training set In some situations  one might not need to distinguish between the different misunder standing categories  NO-RECOG, RMISMATCH and RPARTIAL-MATCH  Therefore experi ments were performed that collapsed these problematic categories into one category RIN-