have codified the notion of a state estimator so that we can systematically vary the state representation in order to explore the effect of the state representation on the value function and the optimal policy   Singh  Kearns  Litman      Walker We are also in the process of using reinforcement learning to conduct a set of experiments on a spoken dialogue system for accessing information about activities in New Jersey  In these experiments we explore a number of different reward functions and also explore a much broader range of strategies for user initiative  for reprompting the user  and for confirming the system s understanding

## 6. Acknowledgements

I received many useful questions and comments on this research when I presented some in the initial results at an invited talk given at AAAI
 in Providence  R I  The design and implementation of the basic functionality of ELVIS was done in collaboration with J  Fromer G  DiFabbrizio  D  Hindle and C  Mestel   Initial experiments on reinforcement learning with ELVIS were done in collaboration with J Fromer and S Narayanan  This work has also benefited from discussions with W  Eckert  C  Kamm  M  Kearns  E  Levin  D  Litman  D
McAllester R  Pieraccini  R  Sutton  and S  Singh  Special thanks are to S  Whittaker  J
Wiebe and four reviewers for detailed comments on earlier versions of this manuscript

## References

Allen J F
A Plan Based Approach to Speech Act Recognition  Tech  rep   Uni versity of Toronto Baggia    P    Castagneri    G
Danieli M
Field trials of the italian arise train timetable system In Interactive Voice Technology for Telecommunications Applications, IVTTA pp Barto A  Bradtke S J
Singh S P
Learning to act using real time dynamic programming   Artificial Intelligence Journal  72 (1-2)
Bellman R E
Dynamic Programming  Princeton University Press  Princeton  N J
Biermann A W
Long P M
The composition of messages in speech graphics interactive systems  In Proceedings of the 1996 International Symposium on Spoken Dialogue pp Bruce B
Belief systems and language understanding  Tech  rep  AI
Bolt Berenak and Newman Carberry S
Plan recognition and its use in understanding dialogue  In Kobsa Wahlster  W   Eds    User Models in Dialogue Systems  pp A
Springer Verlag   Berlin Carbonell J R
Mixed initiative man computer dialogues   Tech  rep Beranek and Newman  Cambridge  MA
Bolt