Recommendation 7: Studies should consider the role of minor syntactic vari ations in performance and include permuted problems ie  initial conditions goals  preconditions and actions  in their problem sets because they can demon strate robustness provide an opportunity for learning and protect developers from accidentally over fitting their algorithm to the set of test problems Problem Assumption 3:  Does Performance Depend on PDDL Requirements Features?  The planners did not perform quite as advertised or expected given some problem features  This discrepancy could have many possible causes  problems incorrectly specified  planners  with less sensitivity than thought  solutions not being correct  etc   For example  many of the problems in the benchmark set were not designed for the competitions or even intended to be widely used and so may not have been specified carefully enough Recommendation 8: When problems are contributed to the benchmark set developers should verify that the requirements stated in the description of each problem correctly reflect the subset of features needed  Planner evaluators should then use only those problems that match a planner s capabilities Depending on the cause the results can be skewed eg a planner may be unfairly maligned for being unable to solve a problem that it was specifically designed not to solve The above recommendation addresses gaps in the specification of the problem set  but some mismatches between the capabilities specifiable in PDDL and those that planners possess remain Recommendation 9:
 Planner developers should develop a vocabulary for their planners capabilities as in the PDDL flags and specify the expected capabilities in the planner s distribution Planner Assumption 1: Is the Latest Version the Best?   Our results suggest that new versions run faster  but often do not solve more problems   Thus  the newest version may not represent the   best"   depending on your definition   performance for the class of planner Some competitions in other fields  e g   the automatic theorem proving community  require the previous year s best performer to compete as well  this has the advantage of establishing a baseline of performance as well as allowing a comparison to how the focus may shift over time Recommendation 10: If the primary evaluation metric is speed  then a newer version may be the best competition If it is number of problems solved or if one wishes to establish what progress has been made  then it may be worth running against an older version as well  If recommendation   has been followed  then evaluators should select a version based on this guidance Planner Assumption 2: What Are the Effect of Parameter Settings?  Perfor mance of some planners does vary with the parameter settings  Unfortunately it often is difficult to figure out how to set the parameters properly and changing settings makes it difficult to compare results across experiments  Generally this is not an issue because the