
| True Values    | Predicted successful | Predicted problematic   | total   |
|----------------|------------------------------------------------|---------|
| TA SK SU CCESS |                                                |         |
| TA SKFAILURE   |                                                |         |
| WIZARD         |                                                |         |
| HANGUP         |                                                |         |
| Total          |                                                |         |

Table Matrix of recognized TASKSUCCESS and TASKFAILURES

| True Values    | Predicted successful | Predicted problematic   | Total   |
|----------------|------------------------------------------------|---------|
| TA SK SU CCESS |                                                |         |
| TA SKFAILURE   |                                                |         |
| Total          |                                                |         |

Table Matrix of recognized TASKSUCCESS and TASKFAILURES using equal training and testing dialogues are more difficult to predict than others we conducted a post hoc analysis of the proportion of prediction failures for each type of problematic dialogue  Since we were primarily interested in the performance of the PDP using the full automatic feature set after having seen Exchanges       we conducted our analysis on this version of the PDP
shows the distribution of the   types of dialogue in the test set and whether the Table Exchanges PDP was able to predict correctly that the dialogue would be TASKSUCCESS
OR PROBLEMATIC  One can see that the worst performing category is TASKFAILURE and that the PDP predicts incorrectly that of the TASKFAILURE dialogues are TASKSUCCESS
One reason that this might occur is that this sub category of dialogues are much more difficult to predict since in this case the HMIHY system has no indication that it is not sseeeding in the task However another possibility is that the PDP performs poorly on this category because there are fewer examples in the training set  although it does better on the HANGUP subset which is about the same proportion  We can eliminate the first possibility by examining how a learner performs when trained on equal proportions of TASK SUCCESS and TASKFAILURE dialogues  We conducted an experiment using a subset of TASKSUCCESS dialogues in the same proportion as TASKFAILURE for the training and the to test set and trained a second PDP using the fully automatic Exchange features  This resulted in a training set of      dialogues and a test set of The binary classifier has an the corresponding recognition matrix is presented in table accuracy of The results show that fewer TASKFAILURES are predicted as successful  suggesting that  TASKFAILURES
are not inherently more difficult to predict than other classes of problematic dialogues Below we discuss the potential of using RIPPER s loss ratio to weight different types of classification errors in future work