developers and other users tend to rely on the default parameter settings  Unfortunately sometimes the developers exploit alternative settings in their own experiments  complicating later comparison Recommendation 11: If a planner includes parameters  the developer should guide users in their settings If they do not then the default settings should be used by both the developers and others in experiments to facilitate comparison Planner Assumption 3:  Are Time Cut-offs Unfair?  We found little benefit from increasing time cut offs beyond minutes for our problems Recommendation 12:  If total computation time is a bottleneck  then run the problems in separate batches  incrementally increasing the time cut off between runs and including only unresolved problems in subsequent runs   When no additional problems are solved in a run  stop Metric Assumption 1:  Do Alternative Platforms Lead to Different Performance?  In our experiments  performance did not vary as mu as we expected  This result suggests that researchers in general are not developing for specific hardware  software configurations  but recent trends suggest otherwise  at least with regards to memory  Again because these systems are research prototypes  it behooves the developer to be clear about his her expectations and anyone subsequently using the system to accommodate those re quests in their studies Recommendation 13: As with other factors in planner design researchers must clearly state the hardware software requirements for their planners  if the design is based on platform assumptions  Additionally  a careful study of mem ory versus time trade offs should be undertaken given the recent trends in mem ory exploitation Metric Assumption 2: Do the Number of Plan Steps Vary?  They certainly can If one neglects quality measures then some planners are being penalized in efforts to declare a best planner Recommendation 14: To expedite generalizing across studies  reports should describe performance in terms of what was solved  how many of what types how much time was required and what were the quality of the solutions  Trade on the offs should be reported when possible electrical increase  in  computation  time decrease in plan length  Additionally  if the design goal was to find an for optimal solution  compare to other planners with that as their design goal Good metrics of plan quality are sorely needed  The latest specification of the PDDL
ssecification supports the definition of problem specific metrics  Fox Long these metrics indicate whether total time  a new concept supported by specification of action durations  or specified functions should be minimized or maximized  This addition is an excellent start  but general metrics other than just plan length and total time are also needed to expedite comparisons across problems