The function q ( e ) represents the cost of transferring control to a particular entity, e.g., the cost of a WAP phone message to a user. Notice, that these detailed, domain-specific costs do not appear directly in the model.

Given the MDP's state space, actions, transition probabilities, and reward function, an agent can use value iteration to generate a policy P : S → Γ that specifies the optimal action in each state  (Puterman, 1994). The agent then executes the policy by taking the action that the policy dictates in each and every state in which it finds itself. A policy may include several transfers of control and coordination-change actions. The particular series of actions depends on the activities of the user. We can then interpret this policy as a contingent combination of many transfer-of-control strategies, with the strategy to follow chosen depending on the user's status (i.e., agent-expect-ρ).

## 4.3  Example: The E-Elves Mdps

An example of an AA MDP is the generic delay MDP , which can be instantiated for any meeting for which Friday may act on behalf of its user. Recall the decision, d, is whether to let other meeting attendees wait for a user or to begin their meeting. The joint activity, a, is the meeting in which the agent has the role, p, of ensuring that its user attends the meeting at the scheduled time. The coordination constraints, ~, are that the attendees arrive at the meeting location simultaneously and the effect of the D action is to delay or cancel the meeting.

In the delay MDP's state representation, team-orig-expect-p is originally-scheduledmeeting-time , since attendance at the originally scheduled meeting time is what the team originally expects of the user and is the best possible outcome. team-expect-o is timerelative-to-meeting, which may increase if the meeting is delayed. α-status becomes statusof-meeting, agent-expect-p is not represented explicitly; instead, user-location is used as an observable heuristic of when the user is likely to attend the meeting. For example, a user who is away from the department shortly before a meeting should begin is unlikely to be attending on time, if at all. With all the state features, the total state space contains 200 states for each individual meeting, with the large number of states arising from a very fine-grained discretization of time.

The general reward function is mapped to the delay MDP reward function in the following way.

$$f_{1}=\left\{\begin{array}{ll}g(N,\alpha)&\mbox{if$N<4$}\\ \infty&\mbox{otherwise}\end{array}\right.\tag{12}$$

where N is the number of times the meeting is rescheduled and g is a function that takes into account factors like the number of meeting attendees, the size of the meeting delay and the time until the originally scheduled meeting time. This function effectively forbids the agent from ever performing 4 or more D actions.

In the delay MDP, the functions, f21 and f22, both correspond to the cost of making the meeting attendees wait, so we can merge them into a single function, f 2 . We expect that such a consolidation is possible in similar domains where the team's expectations relate to