agent programs that are failing due to inconsistency. In theory, Dynamic Hierarchical Justification imposes requirements for handling interruptions that do pose serious questions about its overall utility. In practice, we have not found addressing these questions to be a problem in a variety of recent agent implementations using the Soar-DHJ architecture (e.g., Laird, 2001; Wray et al., 2002).

## 4. Empirical Evaluation Of Dynamic Hierarchical Justification

Architectural mechanisms like DHJ must be efficient. We have demonstrated that the algorithm itself is efficient, but the question of its impact on the overall behavior generation capability of an agent remains an open question due to interruption and regeneration. Given the complexity of both agent-based systems and the domains in which they are applied, analytical evaluations must be extremely narrow in scope, and even then require specialized techniques (Wooldridge, 2000). This section instead pursues an empirical evaluation of Dynamic Hierarchical Justification, focusing on efficiency and responsiveness in two domains at extremes in the continua of agent domain characteristics. Because the architectural solution to inconsistency was motivated by the cost (and incompleteness) of knowledgebased solutions, knowledge development costs will also be estimated.

## 4.1 Methodological Issues

Dynamic Hierarchical Justification is a general solution, applicable in a wide range of agent tasks. In order to evaluate such a solution, a number of methodological issues must be addressed. The following describes three important issues **and the choices made for this** evaluation.

## 4.1.1 Relative Vs. Absolute Evaluation

What constitutes "good" or "poor" cost and performance evaluations? In general, an absolute evaluation of performance and cost is difficult because the task itself, in addition to the agent's knowledge and architecture, determines overall cost and performance results.

We circumvent this problem by making relative comparisons between agents using the original, Fixed Hierarchical Justification Soar architecture ("FHJ agents") and new agents ("DHJ agents"). The FHJ agents provide cost and performance **benchmarks, obviating the** need for absolute evaluations.

## 4.1.2 Addressing Multiple Degrees Of Freedom In Agent Design

Even when architecture and task are fixed, many different functional agents can be developed. How can one know if comparative results are valid and general if the experimenter has control over both benchmarks and new agents?

DHJ agents will be compared to agents previously implemented by others. Such systems will provide good performance targets, because they were optimized for performance, and will minimize bias, because they were developed independently.

FHJ systems were used as fixed benchmarks, and were not modified. DHJ agents use the identical task decompositions employed by the FHJ agents and the same initial knowledge base. We observed opportunities to improve performance in the DHJ agents by modifying