
| Measure            | LMS          | GA           | Boosting       | Off-EEL      | On-EEL          |
|--------------------|--------------|--------------|----------------|--------------|-----------------|
| bcw                |              |              |                |              |                 |
| Train error        | 3.9% (0.2%)  | 1.8% (0.2%)  | 0.0% (0.0%)    | 1.4% (0.2%)  | 0.4% (0.4%)     |
| Test error         | 4.0% (1.6%)  | 3.2% (1.7%)  | 5.3% (2.0%)    | 3.4% (1.7%)  | 3.5% (2.0%)     |
| Test error p-value | 0.00         | -            | 0.00           | 0.09         | 0.04            |
| Ensemble size      | -            | -            | 291.6 (68.2)   | 235.6 (66.8) | 116.3 (278.2)   |
| bld                |              |              |                |              |                 |
| Train error        | 29.8% (0.9%) | 25.4% (1.2%) | 0.0% (0.0%)    | 20.9% (1.5%) | 18.9% (2.0%)    |
| Test error         | 30.4% (6.6%) | 32.7% (6.6%) | 30.4% (5.4%)   | 29.2% (7.4%) | 29.5% (8.4%)    |
| Test error p-value | 0.04         | 0.00         | 0.14           | -            | 0.64            |
| Ensemble size      | -            | -            | 1081.4 (166.1) | 301.0 (37.9) | 294.1 (154.2)   |
| bos                |              |              |                |              |                 |
| Train error        | 32.2% (1.3%) | 23.4% (4.1%) | 0.0% (0.0%)    | 16.7% (1.9%) | 20.9% (2.3%)    |
| Test error         | 34.0% (6.7%) | 30.7% (7.5%) | 26.9% (4.2%)   | 22.7% (5.7%) | 26.2% (7.2%)    |
| Test error p-value | 0.00         | 0.00         | 0.00           | -            | 0.00            |
| Ensemble size      | -            | -            | 761.1 (40.8)   | 303.8 (41.4) | 2960.9 (2109.3) |
| cmc                |              |              |                |              |                 |
| Train error        | 51.6% (0.4%) | 45.7% (1.4%) | 43.3% (0.7%)   | 42.9% (1.2%) | 43.9% (1.4%)    |
| Test error         | 51.8% (2.5%) | 50.4% (3.9%) | 46.8% (2.9%)   | 46.8% (3.9%) | 47.7% (3.9%)    |
| Test error p-value | 0.00         | 0.00         | 0.99           | -            | 0.04            |
| Ensemble size      | -            | -            | 4000.0 (0.0)   | 326.4 (35.7) | 2707.7 (1696.1) |
| pid                |              |              |                |              |                 |
| Train error        | 22.0% (0.6%) | 20.2% (0.7%) | 0.6% (0.5%)    | 19.8% (0.7%) | 20.0% (0.8%)    |
| Test error         | 22.8% (3.5%) | 24.2% (3.9%) | 28.1% (5.0%)   | 24.0% (4.0%) | 24.0% (3.9%)    |
| Test error p-value | -            | 0.00         | 0.00           | 0.00         | 0.00            |
| Ensemble size      | -            | -            | 1978.1 (43.0)  | 309.5 (37.6) | 1196.3 (765.7)  |
| spa                |              |              |                |              |                 |
| Train error        | 11.1% (0.4%) | 7.9% (0.5%)  | 1.4% (0.1%)    | 6.1% (0.2%)  | 7.6% (0.8%)     |
| Test error         | 11.3% (1.2%) | 9.0% (1.3%)  | 5.7% (0.8%)    | 6.7% (1.2%)  | 8.3% (1.4%)     |
| Test error p-value | 0.00         | 0.00         | -              | 0.00         | 0.00            |
| Ensemble size      | -            | -            | 2000.0 (0.0)   | 331.1 (28.4) | 6890.0 (2938.1) |

Table 3: Results on the UCI datasets based on 10-folds cross-validation, using 10 independent runs over each fold. Values are averages (standard deviations) over the 100 runs. Statistical tests are p**-values of paired**
t**-tests on the test error rate compared to that of the best method on the dataset (in bold).** compared to Boosting.

Despite its larger ensemble size, On**-EEL is dominated by**
Off**-EEL on all problems but** pid**, where both approaches**
generate identical test error rates. A tentative explanation stems from the nature of the two approaches, with Off**-EEL** having a clear algorithm organized in two stages, classifiers evolution with diversity-enhancing fitness followed by ensemble construction, while On**-EEL is more complex, with**
a succession of ensemble construction and classifiers evolution with diversity-enforcing measure taken relatively to the current ensemble. The dynamics of On**-EEL is hard** to understand, but it can be speculated that the iterative construction of the ensemble (without individual removal)
is prone to be stuck in local optima. Indeed, the "construction path" taken to build the ensemble begins with a selection of some (supposed poor) individuals at the beginning of the evolution. As these individuals cannot be removed from the ensemble, they significantly influence the choice of other individuals, biasing and possibly misleading the whole process.

## 6. Discussion And Perspectives

This paper has examined the "Evolutionary Ensemble Learning for Free" claim, based on the fact that, since Evolutionary Algorithms maintain a population of solutions, it comes naturally to use these populations as a pool for building classifier ensembles.

Two main issues have been studied, respectively concerned with enforcing the diversity of the population of classifiers, and with selecting the classifiers either in the final population or along evolution.

The use of a co-evolution-inspired fitness function, generalizing [18], was found sufficient to generate diverse classifiers. As already noted, there is a great similarity between the co-evolution of programs and fitness cases [13] and the Boosting principles [8]; the common idea is that good classifiers are learned from good examples, while good examples are generalized by good classifiers. The difference between Boosting and co-evolution is that in Boosting, the training examples are not evolved; instead, their weights are updated. However, the uncontrolled growth of some weights, typically in the case of noisy examples, actually appears as the Achilles' heel of Boosting compared to Bagging. Basically, AdaBoost can be viewed as a dynamic system [22];
the possible instability or periodicity of this dynamic system has undesired consequences on the ensemble learning performance. The use of co-evolutionary ideas, even though the set of ensemble does not evolve, seems to increase the