of the state space, showing only the user-response, and user location features. A transition labeled "delay n" corresponds to the action "delay by n minutes". The figure also shows multiple transitions due to "ask" (i.e., transfer control to the user) and "wait" actions, where the relative probability of each outcome is represented by the thickness of the arrow.  Other state transitions correspond to uncertainty associated with a user's response (e.g., when the agent performs the "ask" action, the user may respond with specific information or may not respond at all, leaving the agent to effectively "wait"). One possible policy produced by the delay MDP, for a subclass of meetings, specifies "ask" in state SO of Figure 5(b)
(i.e., the agent gives up some autonomy). If the world reaches state S3, the policy specifies
"wait". However, if the agent then reaches state S5 , the policy chooses "delay 15", which the agent then executes autonomously. In terms of strategies, this sequence of actions is HD.

Earlier, we described another AA decision in the E-Elves, namely whether to close an auction for an open team role. Here, we briefly describe the key aspects of the mapping of that decision to the MDP. The auction must be closed in time for the user to prepare for the meeting, but with sufficient time given for interested users to submit bids and for the human team leader to choose a particular user. team-orig-expect-ρ(s) is that a highquality presenter be selected with enough time to prepare. There is no D action, hence team-expect-ρ(s) = team-orig-expect-ρ(s). agent-expect-ρ(s) is whether the agent believes it has a high-quality bid or believes such a bid will arrive in time for that user to be allocated to the role. The agent's decision quality, EQ k ( t ), is a function of the number of bids that have been submitted and the quality of those bids, e.g., if all team members have submitted bids and one user's bid stands out, the agent can confidently choose that user to do the presentation. Thus, a -status is primarily the quality of the best bid so far and the difference between the quality of that bid and the second-best bid. The most critical component of the reward function from Equation 10 is the λ 2 component, which gives reward if the agent fulfills the users' expectation of having a willing presenter do a high-quality presentation.

## 4.4 User-Specified Constraints

The standard MDP algorithms provide the agent with optimal policies subject to the encoded probabilities and reward function. Thus, if the agent designer has access to correct models of the entities' (e.g., human users in the E-Elves) decision qualities and probabilities of response, then the agent will select the best possible transfer-of-control strategy. However, it is possible that the entities themselves have more accurate information about their own abilities than does the agent designer. To exploit this knowledge, an entity could communicate its model of its quality of decision and probability of response directly to the agent designer. Unfortunately, the typical entity is unlikely to be able to express its knowledge in the form of our MDP reward function and transition probabilities. An agent could potentially learn this additional knowledge on its own through its interactions with the entities in the domain. However, learning may require an arbitrarily large number of such interactions, all of which will take place without the benefit of the entities' inside knowledge.

As an alternative, we can provide a language of constraints that allows the entities to directly and immediately communicate their inside information to the agent.  Our constraint