Planner Assumption 2: Do Parameter Settings Matter to a Fair Comparison?

In this planner set  only three have obvious  easily manipulable parameters   Blackbox  HSP
and UCPOP  blackbox has an extensive set of parameters that control everything from how much trace information to print to the sequence of solver applications  HSP s function can be varied to include  or not  loop detection  change the search heuristic and vary the number paths to expand  For UCPOP  the user can change the strategies governing node orderings and flaw selection We did not run any experiments for this assumption because not all of the planners have parameters and because it is clear from the literature that the parameters do matter Blackbox relies heavily on random restarts and trying alternative SAT solvers  In Kautz the authors of blackbox carefully study aspects of blackbox s design and and Selman demonstrate differential performance using different SAT solvers they propose hypotheses for the performance differences and are working on better models of performance variation At the heart of HSP is heuristic search  Thus  its performance varies depending on the heuristics Experiments with both HSP and FF a planner that builds on some ideas from HSP  have shown the importance of heuristic selection in search space expansion computation time and problem scale up  Haslum Geffner Hoffmann Ne bel As with HSP heuristic search is critical to UCPOP s performance A set of studies have explored alternative settings to the flaw selection heuristics employed by UCPOP   Joslin Pollack Srinivas an Howe Genevini Schubert producing dramatic improvements on some domains with some heuristics  As Pollack et al cn firmed a good default strategy could be derived  but its performance was not the best under some circumstances Thus  because parameters can control fundamental aspects of algorithms  such as their search strategies  the role of parameters in comparisons cannot be easily dismissed Planner Assumption 3:  Are Time Cut-offs Unfair?  Planners often do not admit to failure Instead the planner stops when it has used the allotted time and not found a solution  So setting a time threshold is a requirement of any planner execution In a comparison  one  might  always  wonder  whether  enough  time  was  allotted  to  be  fair perhaps the solution was almost found when execution was terminated To determine whether our cut off of minutes was fair  we examined the distribution of times for declared successes and failures 8 Across the planners and the problem set  we found that the distributions were skewed  approximately log normal with long right tails and that the planners were quick to declare success or failure  if they were going to do so shows the max_mean_median and standard deviation for success and failure times Table for each of the planners The differences between mean and median indicate the distribution skew  as do the low standard deviations relative to the observed max times  The max time shows that on rare occasions the planners might make a decision within   minutes of our cut off