
## 4.4.3. Results: Hand-Crafted H∗

We re-use the same pHTNs encoding our preferences in Logistics and Gold Miner from the first set of evaluations. As mentioned we use the same setup as in the random experiments, so it continues to be the case that the distribution on random 'solutions' is biased against the encoded preferences. Moreover, due to the level of abstraction used (truncating to action names), as well as the nature of the pHTNs and domains in question, the randomly generated sets of alternatives, Fi, are in fact sets of solutions to *some* problem expressed in the normal fashion
(i.e., as an initial state and goal).

Logistics. After training with 550 training records (50t, for 11 non-primitives)
the baseline system scored only 0.342 (0 is the performance of random guessing)
whereas rescaling before learning performed significantly better with a score of 0.847 (0.153 away from perfect performance).

Gold Miner: After training with 600 examples (50t for 12 non-primitives) learning alone scored a respectable 0.605, still, rescaling before learning performed better with a score of 0.706. Note that the greater recursion in Gold Miner, as compared to Logistics, is both hurting and helping. On the one hand the full approach scores worse (0.706 vs. 0.847), on the other hand, the baseline's performance is hugely improved (0.605 vs. 0.342). As discussed previously, the presence of recursion in the preference model makes the learning problem much harder (since the space of acceptable plans is then actually infinite), which continues to be a reasonable explanation of the first effect (degrading performance).

The latter effect is more subtle. The experimental setup, roughly speaking, inverts the probability of selecting a plan, so that using a recursive method many times in an observed plan is more likely than using the same method only a few times. Then the baseline approach is attempting to learn a distribution skewed towards greater use of recursion overall, and in particular, a distribution that prefers more recursion to less recursion all else being equal. However, there is no pHTN that prefers more recursion to less recursion all else being equal; fewer uses of a recursive method always increases the probability of a plan. So the baseline will fit an inappropriately large probability to any recursive method, but, it will still make the correct decision between two plans differing only in the depth of their recursion over that method. Naive Bayes Classifiers exhibit a similar effect [9, Box 17.A].