- Was ELVIS easy to understand in this conversation   TTS Performance
- In this conversation did ELVIS understand what you said   ASR Performance
- In this conversation was it easy to find the message you wanted Task Ease - Was the pace of interaction with ELVIS appropriate in this conversation Interaction Pace
- In this conversation  did you know what you could say at each point of the dialogue User Expertise
- How often was ELVIS sluggish and slow to reply to you in this conversation System Response
- Did ELVIS work the way you expected him to in this conversation    Expected Behavior
- In this conversation how did ELVIS s voice interface compare to the touch tone interface to voice mail    Comparable Interface G From your current experience with using ELVIS to get your email do you think you d use ELVIS regularly to access your mail when you are away from your desk Future Use

## Figure User Satisfaction Survey

In order to calculate User Satisfaction  users were asked to evaluate the system s perfor to mance with the user satisfaction survey in Figure Some of the question responses were on a five point Likert scale and some simply required yes, no or yes, no, maybe responses The survey questions probed a number of different aspects of the users  perceptions of their interaction with ELVIS in order to focus the user on the task of rating the system as in Shriberg et al Jack Foster Stentiford Love Dutton Foster Jack Stentiford Each multiple choice survey response was mapped into the range of to Then the values for all the responses were summed  resulting in a User Satisfaction measure for each dialogue with a possible range of   to

## 4. Training And Testing An Optimized Dialogue Strategy

Given the experimental training data  we first apply PARADISE to estimate a performance function for ELVIS as a linear combination of the metrics described above  We apply the performance function to each dialogue in the training corpus to estimate a utility for the final state of the dialogue and then apply    learning using this utility  Finally we test the learned policy against a new population of users

## Paradise  Perfor Mance   Mode Ling

The first step in developing a performance model for spoken dialogue systems was the spec in ification of the causal model of performance illustrated in Figure Walker et al According to this model  the system's primary objective is to maximize user satisfaction