
## 6.3 Results

The statistics over 10 independent runs can be seen on Figure 1. Three criteria can be used to compare the performances of the 4 architectures: the best overall performance, the variance of the results, and how rapidly good performances are obtained. The sensitivity and generalization abilities of the resulting controllers are important criteria that require additional experiments (sections 6.4, 6.5).

The best overall performance are obtained by the SS (Symbolic Supervisor)
architecture. Moreover, it exhibits a very low variance (average best fitness is 6442 ± 28). Note that overpassing a fitness of 6400 means that the resulting behavior could go on for ever, almost optimally storing fitness between **the recharge**
phases).

Next best architecture is the CC (Classical Controller). But whereas its best overall fitness is only slightly less that that of the SS, the variance is **10 times** larger (average best fitness is 6044 ± **316, with best at 6354). The difference is** statistically significant with 95% confidence using the Student T-test.

The SC (Symbolic Controller) and CS (Classical Supervisors) come last, with respective average best fitness of 5902 ± 122 and 5845 ± 27.

Some additional comments can be made about those results. First, **both**
supervisors architectures exhibit a very good best fitness (≈ **3200) in the initial** population: such fitness is in fact obtained when the supervisors only use the obstacle avoidance - they score maximum fitness until their accumulator level goes to 0. Of course, the direct controller architectures require **some time to**
reach the same state (more than 2000 evaluations).

Second, the variance for both supervisor architectures is very low. Moreover, it seems that this low variance is not only true at the performance level, but also at the behavior level: whereas all symbolic supervisors do explore the environment until their energy level becomes dangerously low, and then head toward the light and stay **in the recharge area until their energy level is maximal again,**
most (but not all) of the direct controller architectures seem to simply stay close to the recharge area, entering it randomly.

One last critical issue is the low performance of the Symbolic Controller.

A possible explanation is the existence of the neutrality plateaus discussed in section 3.2: though those plateaus help escaping local minima, they also slow down the learning process. Also it appears clearly on Figure 1-left that the SC architecture is the only one that seems to steadily increase its best **fitness until** the very end of the runs. Hence, the experiment was carried on for another 250 generations, and indeed the SC architecture did continue to improve (over a fitness level of 6200) - while all other architectures simply stagnates.

The evolved behaviors have been further examined. Figure 2-left shows a typical plot of the number of calls of each basic behaviors by the best evolved Symbolic Supervisor during one fitness evaluation. First, it appears **that both**
supervisors architectures mainly use the obstacle avoidance **behavior, never use**
the area sweeping**, and, more surprisingly, almost never use the** light following: