only slightly greater  Thus we implemented the learned policy  see Figure  
with the exception that the Summarize System strategy was used throughout the dialogue 11 In terms of the operations state machine in Figure implementation of the learned policy means that the choices between the SI Top and MI Top strategies are replaced by the SI Top strategy choices between the different read strategies in different states are replaced by the Read First RF strategy and choices between the different summarization strategies in different states are replaced by the Summarize System SS strategy We then tested this policy on six new users who had never used E LVIS before These users conversed with ELVIS to perform the same set of six email tasks that were used tin the training phase as described in Figure above  In addition  identical performance measures were collected for each testing dialogue and training dialogue   Overall performance measures for the training and test dialogues are given in Table - with the training data split in terms of System Initiative  Mixed Initiative and overall means  The table shows that all versions of ELVIS have high levels of task completion  which is important for testing the utility of reinforcement learning   Statistical analysis of these results indicated a statistically significant increase in User Satisfaction from training to test  F

## 5. Discussion And Future Work

This paper proposes a novel method by which a dialogue system can learn to choose an optimal dialogue strategy and tests it in experiments with ELVIS a dialogue system that supports access to email by phone  with strategies for initiative  and for reading and summa rizing messages We reported experiments in which ELVIS learned that the System Initiative strategy has higher utility than the Mixed Initiative strategy that Read First is the best read strategy and that Summarize System is generally the best summary strategy We then tested the policy that ELVIS learned on a new set of users performing the same set of tasks and showed that the learned policy resulted in a statistically significant increase in user satisfaction in the test set of dialogues Previous work has also treated a system s choice of dialogue strategy as a stochastic otimization problem  Walker Biermann L ong Levin Pieraccini To our knowledge  Walker Levin et al first proposed that reinforcement learning algorithms could be applied to dialogue strategy selection In simulation experi ments reported by Walker dialogues between two agents in an artificial world were used to test which dialogue strategies were optimal under various conditions  These the dialogue agent s resource bounds  and  
the performance func experiments varied tion used to assess the agent s performance  The experiments showed that strategies that were not optimal under one set of assumptions about the performance function could be highly efficacious when the performance function reflected the fact that the dialogue agent was resource bounded  Walker suggested that the optimal dialogue strategy could be