expect that users of an email system would engage in many interactions with the same system  and that preferences for system interaction strategies could change over time with user expertise This means that the performance function might change over time We also used a fixed set of tasks that were representative of the domain  but it is possible that some aspects of the policies we learned might be sensitive to our experimental tasks Another limitation is that the experiments were carried out in a scenario where each email folder only had a small number of messages  the strategies tested here might not be optimal when an email folder contains hundreds of messages Fourth  the optimal strategy is potentially dependent on various system parameters   For example the ReadFirst strategy takes the initiative to read a message  which might result in messages being read that the user wasn t interested in  but since the user can barge in on system utterances  there is little overhead with taking this decision  If the system did not support barge in  our results might have been different Fifth  the learned policy depends on the reward function  For example  since ELVIS
is a fully functional system  users can complete the experimental task with any version of the system using any of the strategies that we explored This means that if we had used task completion as the reward function  reinforcement learning would have predicted that there were no differences between the different strategies  On the other hand  by using the PARADISE performance function  we utilized a reward function that was fit to our data and ELVIS s performance  and we have some evidence that this reward function may generalize to other systems  Walker  Kamm     Litman Sixth the experiments that we report here are limited in the way that they demonstrate the utility of reinforcement learning for dialogue strategy optimization A more traditional way of selecting the best dialogue strategies would be with experiments which treated dialogue strategy selection as a factor  and standard statistical hypothesis testing would be used to compare the performance of different strategies   The scale of the experiment here is small enough that it is imaginable that the space of policies could possibly have been tested in the more traditional way  However  the primary goal of the experiments reported here was simply to test the feasibility of these methods  which required working out in detail many of the issues of state and strategy representation discussed above  Now that many of these details have been worked out  the methods presented here can be applied to much more complex dialogue strategy optimization problems  such as varying the initiative depending on the dialogue state  Chu Carroll    Brown Webber Joshi exploring combinations of strategies for information presentation  summarization  Sparck error recovery  Hirschman   Pao database query   Levin et al Jones ccooperative responses   Joshi   Webber Weischedel Finin et al Chu Carroll Carberry and content selection for generation   McKeown Kittredge Korelsky Rambow inter alia Finally the learning algorithm that we report here is an off-line algorithm ie ELVIS
collects a set of dialogues and then decides on an optimal strategy as a result In contrast it should be possible for Elvis to learn on-line during the course of a dialogue  once methods are developed for the performance function to be automatically calculated or approximated Our primary goal with the experiments reported here was to explore the application of reinforcement learning to spoken dialogue systems and to identify open issues such as those discussed above In current work we are exploring these issues in several ways We