others  they were able to demonstrate performance of their planner on well known prob lems  which could be viewed as de facto benchmarks  Sussman s anomaly  Sussman in Blocksworld was the premier planning benchmark problem and domain for many years every planner needed to  cut its teeth" on it As researchers tired of Blocksworld many called for additional benchmark problems and environments  Mark Drummond  Leslie Kaelbling and Stanley Rosenschein organized a workshop on benchmarks and metrics  Drummond  Kaelbling Rosenschein Testbed environments such as Martha Pollack s TileWorld Pollack Ringuette or Steve  Hanks s  TruckWorld   Hanks   Nguyen Thomas were used for comparing agorithms within planners   By UCPOP Penberthy Weld was distributed with a large set of problems problems in     domains  for demonstration purposes  In Barry Fox and Mark Ringer set up a planning and scheduling benchmarks web page http://www.newosoft.com/~benchmrx/ to collect problem definitions  with an emphasis on manufacturing applications  Recently  PLANET  a coordinating organization for Euro pean planning and scheduling researchers  has proposed a planning benchmark collection initiative  http   planet dfki de Clearly benchmark problems have become well established means for demonstrating planner performance  However  the practice has known benefits and pitfalls  Hanks  Pollack discuss them in some detail in the context of agent architecture design and Cohen The benefits include providing metrics for comparison and supporting experimental control The pitfalls include a lack of generality in the results and a potential for the benchmarks to unduly influence the next generation of solutions  In other words  researchers will construct solutions to excel on the benchmarks  regardless of whether the benchmarks accurately represent desired real applications To obtain the benefits just listed for benchmarks the problems often are idealized or simplified versions of real problems  As Cohen points out  most research papers in AI or at least at an AAAI conference exploit benmark problems  yet few of them relate the benchmarks to target tasks  This may be a significant problem  for example  in a study of flowshop scheduling 2 benchmarks  we found that performance on the standard bench mark set did not generalize to performance on problems with realistic structure  Watson A study of just Blocksworld problems found that the Barbulescu   Howe Whitley best known Blocksworld benchmark problems are atypical in that they require only short to plans for solution and optimal solutions are easy to find  Slaney Thieb aux In spite of these difficulties  benchmark problems and the AIPS competitions have con siderably influenced comparative planner evaluations  For example  in the AIPS
con fference proceedings  Chien  Kambhampati Knoblock all of the papers on im provements to classical planning     out of papers at the conference  relied heavily on comparative evaluation using benchmark problems  the other papers concerned scheduling specific applications theoretical analyses or special extensions to the standard paradigm eg  POMDP  sensing   Of the    classical papers  six used problems from the AIPS
competition benchmark set six used problems from Kautz and Selman's distribution of problems with blackbox  Kautz and three added some of their own problems as well Each paper showed results on a subset of problems from the benchmark distributions