stability of the learning process.

The two EEL frameworks investigated in this paper can be considered as promising. Off**-EEL constructs ensembles** with best performances while needing little modifications over a traditional evolutionary algorithm, with a diversityenhancing fitness and the construction of an ensemble from the final population. But the size of the ensembles generated suggests that bigger population would lead to bigger and possibly better ensembles. For the sake of scalability, this suggests that the ensemble should be gradually constructed along evolution, instead of considering only the final population. This has been explored with On**-EEL, with**
lesser performance comparing to Off**-EEL. It is suggested**
that ensemble construction with On**-EEL is prone to be**
stuck in local minima, so some capability of removing individuals can be beneficial, at the risk of inducing an highly dynamic algorithm. Ultimately, the momentum and dynamics of EEL should be controlled by evolution itself, enforcing some trade-off between exploring new regions and preserving efficient optimization. This will be the subject of future researches.

## Acknowledgments

This work was supported by postdoctoral fellowships from the ERCIM-SARIT (Europe), the Swiss National Science Foundation (Switzerland), and the FQRNT (Qu´ebec) to C.

Gagn´e. The second and third authors gratefully acknowledge the support of the Pascal **Network of Excellence IST2002-506 778.**

## 7. References

[1] L. Breiman. Arcing classifiers. **Annals of Statistics**,
26(3):801–845, 1998.

[2] A. Chandra and X. Yao. Ensemble learning using multi-objective evolutionary algorithms. **J. of**
Mathematical Modelling and Algorithms**, 5(4):417–425,**
2006.

[3] A. Chandra and X. Yao. Evolving hybrid ensembles of learning machines for better generalisation.

Neurocomputing**, 69:686–700, 2006.**
[4] T. G. Dietterich. Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation**, 10:1895–1923, 1998.**
[5] T. G. Dietterich. Ensemble methods in machine learning. In **First Int. Workshop on Multiple Classifier**
Systems**, pages 1–15, 2000.**
[6] R. Esposito and L. Saitta. Monte Carlo theory as an explanation of Bagging and Boosting. In **Proc. of the** Int. Joint Conf. on Artificial Intelligence (IJCAI'03),
pages 499–504, 2003.

[7] G. Folino, C. Pizzuti, and G. Spezzano. Ensemble techniques for parallel genetic programming based classifiers. In **Proc. of the European Conf. on Genetic** Programming (EuroGP'03)**, pages 59–69, 2003.**
[8] Y. Freund and R. Schapire. Experiments with a new Boosting algorithm. In **Proc. of the Int. Conf. on** Machine Learning (ICML'96)**, pages 148–156, 1996.**
[9] C. Gagn´e and M. Parizeau. Genericity in evolutionary computation software tools: Principles and case-study.

Int. J. on Artificial Intelligence Tools**, 15(2):173–194,**
2006.

[10] C. Gagn´e and M. Parizeau. Open BEAGLE: An evolutionary computation framework in C++.

http://beagle.gel.ulaval.ca**, 2006.**
[11] R. Gilad-Bachrach, A. Navot, and N. Tishby. Margin based feature selection - theory and algorithms. In Proc. of the Int. Conf. on Machine Learning
(ICML'04)**, pages 43–50, 2004.**
[12] I. Guyon, S. Gunn, M. Nikravesh, and L. Zadeh, editors. **Feature Extraction: Foundations And**
Applications**. Springer-Verlag, 2006.**
[13] W. D. Hillis. Co-evolving parasites improve simulated evolution as an optimization procedure. **Physica D**,
42:228–234, 1990.

[14] J. Holland. Escaping brittleness: The possibilities of general-purpose learning algorithms applied to parallel rule-based systems. In **Machine Learning, An** Artificial Intelligence Approach**, volume 2, pages** 593–623. Morgan Kaufmann, 1986.

[15] J. Holmes, P. Lanzi, W. Stolzmann, and S. Wilson.

Learning classifier systems: New models, successful applications. **Information Processing Letters**, 82(1):23–30, 2002.

[16] H. Iba. Bagging, Boosting, and bloating in genetic programming. In **Proc. of the Genetic and** Evolutionary Computation Conference (GECCO'99),
pages 1053–1060, 1999.

[17] M. Keijzer and V. Babovic. Genetic programming, ensemble methods, and the bias/variance tradeoff –
introductory investigations. In **Proc. of the European**
Conf. on Genetic Programming (EuroGP'00)**, pages**
76–90, 2000.

[18] Y. Liu, X. Yao, and T. Higuchi. Evolutionary ensembles with negative correlation learning. **IEEE** Trans. on Evolutionary Computation**, 4(4):380–387,** 2000.

[19] D. Newman, S. Hettich, C. Blake, and C. Merz. UCI repository of machine learning databases.

http://www.ics.uci.edu/~mlearn/MLRepository.html, 1998.

[20] J. Paredis. Coevolving cellular automata: Be aware of the Red Queen! In **Proc. of the Int. Conf. on Genetic** Algorithms (ICGA'97)**, pages 393–400, 1997.**
[21] G. Paris, D. Robilliard, and C. Fonlupt. Applying Boosting techniques to genetic programming. In Artificial Evolution 2001, volume 2310 of LNCS**, pages**
267–278. Springer Verlag, 2001.

[22] C. Rudin, I. Daubechies, and R. E. Schapire. The dynamics of AdaBoost: Cyclic behavior and convergence of margins. **J. of Machine Learning**
Research**, 5:1557–1595, 2004.**
[23] R. Schapire, Y. Freund, P. Bartlett, and W. Lee.

Boosting the margin: A new explanation for the effectiveness of voting methods. **Annals of Statistics**,
26(5):1651–1686, 1998.

[24] D. Song, M. I. Heywood, and A. N. Zincir-Heywood.

Training genetic programming on half a million patterns: an example from anomaly detection. **IEEE** Trans. on Evolutionary Computation**, 9(3):225–239,**
2005.

[25] V. N. Vapnik. Statistical Learning Theory**. Wiley, New**
York, NY (USA), 1998.