mand action s description required that a drive be empty and had the effect of changing the position of the tape being mounted and changing the status of the tape drive Problems described initial states of the tape library eg where tapes were resident what was the status of the devices and software controller  and goal states that a human operator might wish to achieve At the time  we found that only the simplest problems could be generated using the planners available  We included this application in part because we knew it would be a challenge As part of the test set we include three domain theories different ways of ol coding the application involving operators  and twenty four problems for each domain We included only because we wanted to include enough problems to see some effect  but not too many to overly bias the results  These problems were relatively simple  requiring the movement of no more than one tape coupled with some status changes  but they were still more difficult than could be solved in our original system

## 3.2 Problem Assumptions

General purpose planners exhibit differential capabilities on domains and sometimes even problems within a domain  Thus  the selection of problem set would seem to be critical to evaluation For example many problems in benchmark sets are variants of logistics problems thus a general purpose planner that was actually tailored for logistics may appear to be better overall on current benchmarks  In this section  we will empirically examine some possible problem set factors that may influence performance results

## Problem Assumption 1: To What Extent Is Performance Of General Purpose

Planners Biased Toward Particular Problems/Domains?   Although most planners are developed as general purpose  the competitions and previous studies have shown that planners excel on different domains problems  Unfortunately the community does not yet have a good understanding of why a planner does well on a particular domain  We studied the impact of problem selection on performance in two ways First we assessed whether performance might be positively biased toward problems tested during development Each developer 6 was asked to indicate which domains they used during development We then compared each planner s performance on their development problems i e  the development set  to the problems remaining in the complete test set We r a n  x
² tests comparing number of problems solved versus failed in the rest development and test sets We included only the number solved and failed in the analysis as timed out problems made no difference to the results 7 The results of this analysis are summarized in Table Figure   graphically displays the ratio of successes to failures for the development and other problems  All of the planners except C performed significantly better on their development problems  This suggests that these planners have been tailored  intentionally or not  for particular types of problems and that they will tend to do better on test sets biased accordingly  For example  one of the