in Table 5, this strategy has a very low EU. When a fixed-length timeout was introduced, Friday would follow strategy e(5)A. Such a strategy has high EU when EQH(t) ~ EQH(t)
but very low EU when EQ (t) ≈ 10 * EQ A (t). Thus, the model explains a phenomenon observed in practice.

On the other hand, we can use the model to understand that C4.5's failure in this case does not mean that it will never be useful for AA. Different strategies are only required when certain parameters (like probability of response or wait cost) change significantly. In applications where such parameters do not change dramatically from decision to decision, one particular strategy may always be appropriate. For such applications, C4.5 might learn the right strategy just with a small amount of training data and perform acceptably well.

## Operationalizing Strategies With Mdps 4.

We have formalized the problem of AA as the selection of the transfer-of-control strategy with the highest EU. We now need an operational mechanism that allows an agent to perform that selection. One major conclusion from the previous section is that different strategies dominate in different situations, and that applications such as E-Elves will require mechanism(s) for selecting strategies in a situation-sensitive fashion. In particular, the mechanism must flexibly change strategies as the situation changes.  The required mechanism must also represent the utility function specified by our expected decision qualities, EQ, the costs of violating coordination constraints, W, and our coordination-change cost, Dcost . Finally, the mechanism must also represent the uncertainty of entity responses and then look ahead over the possible responses (or lack thereof) that may occur in the future.

MDPs are a natural means of performing the decision-theoretic planning required to find the best transfer-of-control strategy. MDP policies provide a mapping between the agent's state and the optimal transfer of control strategy.  By encoding the parameters of the model of AA strategies into the MDP, the MDP effectively becomes a detailed implementation of the model and, hence, assumes its properties. We can use standard algorithms  (Puterman, 1994) to find the optimal MDP policy and, hence, the optimal strategies to follow in each state.

To simplify exposition, as well as to illustrate the generality of the resulting MDP, this section describes the mapping from AA strategies to the MDP in four subsections. In particular, Section 4.1 provides a direct mapping of strategies to an abstract MDP. Section 4.2 fills in state features to enable a more concrete realization of the reward function, while still maintaining a domain-independent view. Thus, the section completely defines a general MDP for AA is potentially reusable across a broad class of domains. Section 4.3 illustrates an implemented instantiation of the MDP in E-Elves.  Section 4.4 addresses further practical issues in operationalizing such MDPs in domains such as E-Elves.

## 4.1  Abstract Mdp Representation Of Aa Problem

Our MDP representation's fundamental state features capture the state of control:
- controlling-entity is the entity that currently has decision-making control.

- e i -response is any response e i has made to the agent's requests for input.