# An Application Of Reinforcement Learning To Dialogue Strategy Selection In A Spoken Dialogue System For Email

## Marilyn A  Walker

WALKER@RESEARCH.ATT.COM
AT&T Shannon Laboratory 180 Park Ave., Bldq 103, Room E103 Florham Park, NJ 07932

## Abstract

This paper describes a novel method by which a spoken dialogue system can learn to choose an optimal dialogue strategy from its experience interacting with human users The method is based on a combination of reinforcement learning and performance model ing of spoken dialogue systems  The reinforcement learning component applies Q learning Watkins while the performance modeling component applies the PARADISE eval uation framework  Walker et al to learn the performance function  reward  used in reinforcement learning We illustrate the method with a spoken dialogue system named ELVIS EmaiL Voice Interactive System that supports access to email over the phone We conduct a set of experiments for training an optimal dialogue strategy on a corpus of dialogues in which human users interact with EIVIS over the phone We then test that strategy on a corpus of     dialogues  We show that EIVIS can learn to optimize its strategy selection for agent initiative for reading messages and for summarizing email folders

## 1. Introduction

In the past several years it has become possible to build spoken dialogue systems that can communicate with humans over the telephone in real time  Systems exist for tasks such as finding a good restaurant nearby reading your email perusing the classified advertisements about cars for sale or making travel arrangements  Seneff  Zue  Polifroni  Pao  Hether ington   Goddeau Baggia  Castagneri Sanderman   Sturm Glass Danieli dn Os  Boves Cremers Walker  Fromer Narayanan These systems are some of the few realized examples of real time goal oriented interactions between humans and computers  Yet in spite of    years of research on algorithms for dialogue manage ment in task oriented dialogue systems   Carbonell Winograd Simmons Slocum Bruce Power Walker Allen Cohen Pollack Hirschberg Webber Grosz Woods Webber Finin Joshi Carberry Moore Paris Smith Hipp Kamm inter alia the design of the dialogue manager in real time  implemented systems is still more of an art than a science  Sparck Jones    Galliers This paper describes a novel method  and experiments that validate the method  by which a spoken dialogue system can learn from its experience with human users to optimize its choice of dialogue strategy The dialogue manager of a spoken dialogue system processes the user s utterance and then chooses in real time what information to communicate to the human user and how to communicate it  The choice it makes is called its strategy  The dialogue manager can be naturally formulated as a state machine  where the state of the dialogue is defined by a set