

![0_image_0.png](0_image_0.png)

## 3.4 Performance Metrics

Most comparisons emphasize the number of problems solved and the CPU time to comple tion as metrics  Often  the problems are organized in increasing difficulty to show scale up Comparing based on these metrics leaves a lot open to interpretation  For example  some planners are designed to find the optimal plan  as measured by number of steps in either a parallel or sequential plan  Consequently these planners may require more computation Thus  by ignoring plan quality  these planners may be unfairly judged  We also hypothesize that the hardware and software platform for the tests can vary the results  If a planner is developed for a machine with  GB of memory  then likely its performance will degrade with less A key issue is whether the effect is more or less uniform across the set of planners In this section we examine these two issues execution platform and effect of plan quality Metric Assumption 1: Does Performance Vary between Planners When Run on Different Hardware Platforms?  Often when a planner is run at a competition or in someone else s lab  the hardware and software platforms differ from the platform used during development Clearly slowing down the processor speed should slow down planning requiring higher cut offs  Reduction in memory may well change the set of problems that can be solved or increase the processing time due to increased swapping  Changing the hardware configuration may change the way memory is cached and organized  favoring some planners internal representations over others  Changing compilers could also affect the amount and type of optimizations in the code  The exact effects are probably unknown The assumption is that such changes affect all planners more or less equally To test this we r an the planners on a less powerful lower memory machine and compared the results on the two platforms  the base Sun Ultrasparc mb of memory with and Ultrasparc with    mb of memory  The operating system and compilers were the same versions for both machines  The same problems were run on both platforms  We followed much the same methodology as in the comparison of planner versions  comparing to on both number of problems solved and time to solution  Table shows the results as measured by problems solved  failed or timed out for each planner on the two platforms