a good choice for problems with conditional effects as it was able to solve many of these problems  This would be a mistake since that planner cannot actually handle these types of problems  In these cases  the problems claim to require ADL  but in fact  they only make use of the STRIPS subset Clearly certain problems can only be solved by specific planners  For instance  C and M are the only planners that are able to handle safety constraints  while based on the data only C D and E appear to handle domain axioms  About half the planners had trouble with the typed problems Some of the gaps appear to be due to problems in the translation to native representation

## 3.3 Planners

Publicly available general purpose planners tend to be large programs developed over a period of years and enhanced to include additional features over time Thus several versions are likely to be available  and those versions are likely to have features that can be turned on off via parameter settings When authors release later versions of their planning systems  the general assumption is that these newer versions will outperform their predecessors  However  this may not be the case in practice  For instance  a planner could be better optimized toward a specific class of problem which then in turn hurts its performance on other problems  Also  advanced capabilities  even when unused  may incur overhead in the solution of all problems So for comparison purposes should one use the latest version First we tested this question in a study comparing multiple versions of four of the planners  Second  each planner relies on parameter settings to tune its performance  Some  such as  blackbox  have many parameters Others have none Comparisons tend to use the default or published parameter settings because few people usually understand the effects of the parameters and tuning can be extremely time consuming  So does this practice undermine a fair cmparison Planner Assumption 1: Is the Latest Version the Best?   In this study  we compared performance of multiple versions of four planners  labeled for this section with W  X  Y and Z  with larger  version numbers indicating subsequent versions    We considered two criteria for improvement  outcome of planning and computation time for solved problems  The outcome of planning is one of  solved  failed or timed out   On each criterion  we statistically analyzed the data for superior performance of one of the versions  The outcome results for all the planners are summarized in Table As the table shows rarely does a new version result in more problems being solved  Only Z improved the number of our test problems solved in subsequent versions To check for whether the differences in outcome are significant  we r an  x     2 tests with planner version as independent variable and outcome as dependent  Table summarizes the results of the 2 analysis For Z we compared each version to its successor only The to differences are significant except for Y and the transition from Z to this was expected because these two versions were extremely similar Another planner performance metric  which we evaluated  was the speed of solution   For this analysis  we limited the comparison to just those problems that were solved by both versions of the planner We then classified ea problem by whether the later version solved