into problem types and key characteristics  Developers should contribute appli cation problems and realistic versions of them to the evolving set The remainder of this section describes other recommendations for improving the state of the art in planner comparisons Problem Assumption 1: Are General Purpose Planners Biased Toward Particular Problems/Domains?  The set of problems on which a planner was developed can have a strong effect on the performance of the planner  This can be either the effect of unintentional over specialization or the result of a concerted effort on the part of the developers to optimize their system to solve a specific problem With one exception every planner fared better on the tailored subset of problems  training set   Consequently  we must conclude that the choice of a subset of problems may well affect the outcome of any comparison A fair planner comparison must account for likely biases in the problem set  Good performance on a certain class of problems does not imply good performance in general A large performance differential for planners with a targeted problem domain  i e  do well on their focus problems and poorly on others  may well indicate that the developers have succeeded in optimizing the performance of their planner Recommendation 4: Problem sets should be constructed to highlight the designers  expectations  about superior performance for their planner  and they should be specific about this selection criteria On the other hand if the goal is to demonstrate across the board performance  then our results at randomly selecting domains suggests that biases can be mitigated Recommendation 5:  If highlighting performance on  general" problems is the goal  then the problem set should be selected randomly from the benchmark domains Problem Assumption 2: How Do Syntactic Representation Differences Affect Performance?

Many studies  including this  have shown that planners may be sensitive to representational features  Just because representations can be translated automatically does not mean that performance will be unaffected  Just because an algorithm should theoretically be insensitive to a factor does not mean that in practice it is  All of the planners showed some sensitivity to permuted problems  and the degree of sensitivity varied This outcome suggests that translators and even minor variations on problem descriptions impact outcome and should be used with care  especially when the sensitivity is not the focus of the study and some other planner is more vulnerable to the effect Recommendation 6: Representation translators should be avoided by using ntive versions of problems and testing multiple versions of problems if necessary With many planner developers participating in the AIPS competitions  this should become less of an issue More importantly  researchers should be explicitly testing the effect of alternative phras ings of planning problems to determine the sensitivity of performance and to separate the effects of advice tuning from the essence of the problem