# Ensemble Learning For Free With Evolutionary Algorithms ?

Christian Gagn´e
∗
Informatique WGZ Inc.,
819 avenue Monk, Qu´ebec (QC), G1S 3M9, Canada.

christian.gagne@wgz.ca

## Abstract

Evolutionary Learning proceeds by evolving a population of classifiers, from which it generally returns (with some notable exceptions) the single best-of-run classifier as final **result. In the meanwhile, Ensemble Learning, one of the most**
efficient approaches in supervised Machine Learning for the last decade, proceeds by building a population of diverse classifiers. Ensemble Learning with Evolutionary Computation thus receives increasing attention. The Evolutionary Ensemble Learning (EEL) approach presented in this paper features two contributions. First, a new fitness function, inspired by co-evolution and enforcing the classifier diversity, is presented. Further, a new selection criterion based on the classification margin is proposed. This criterion is used to extract the classifier ensemble from the final population only (Off-EEL) or incrementally along evolution (On**-EEL).**
Experiments on a set of benchmark problems show that Off**-EEL outperforms single-hypothesis evolutionary learning and state-of-art Boosting and generates smaller classifier**
ensembles.

## Categories And Subject Descriptors

I.5.2 [Pattern Recognition]: Design Methodology—Classifier design and evaluation; I.2.8 [**Artificial Intelligence**]:
Problem Solving, Control Methods, and Search—**Heuristic**
methods

## General Terms Algorithms

∗**This work has been mainly realized during a postdoctoral**
fellowship of Christian Gagn´e at the University of Lausanne.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. GECCO'07, July 7–11, 2007, London, England, United Kingdom.

Copyright 2007 ACM 978-1-59593-697-4/07/0007 ...$5.00.

Mich`ele Sebag Equipe TAO - CNRS UMR 8623 / INRIA Futurs, ´
LRI, Bat. 490, Universit´e Paris Sud, F-91405 Orsay Cedex, France.

michele.sebag@lri.fr Marc Schoenauer Equipe TAO - INRIA Futurs / CNRS UMR 8623, ´
LRI, Bat. 490, Universit´e Paris Sud, F-91405 Orsay Cedex, France.

## Marc.Schoenauer@Lri.Fr

Marco Tomassini Information Systems Institute, Universit´e de Lausanne, CH-1015 Dorigny, Switzerland.

marco.tomassini@unil.ch

## Keywords

Ensemble Learning, Evolutionary Computation

## 1. Introduction

Ensemble Learning, one of the main advances in Supervised Machine Learning since the early 90's, relies on: i) a weak learner (extracting hypotheses, aka classifiers, with error probability less than 1/2 − ǫ, ǫ > **0); ii) a diversification**
heuristics used to extract sufficiently diverse classifiers; **iii)**
a voting mechanism, aggregating the diverse classifiers constructed [1, 8]. If the classifiers are sufficiently diverse and their errors are independent, then their majority vote will reach an arbitrarily low error rate on the training set as the number of classifiers increases [6]. Therefore, up to some restrictions on the classifier space [25], the generalization **error**
will also be low1.

The most innovative aspect of Ensemble Learning w.r.t.

the Machine Learning literature concerns the diversity requirement, implemented through parallel or sequential heuristics. In Bagging, diversity is enforced by considering independent sub-samples of the training set, and/or using different learning parameters [1]. Boosting iteratively constructs a sequence of classifiers, where each classifier focuses on the examples misclassified by the previous ones [8].

Diversity is also a key feature of Evolutionary Computation (EC): in contrast with all other stochastic optimization approaches, evolutionary algorithms proceed by evolving a population of solutions, and the diversity thereof has been stressed as a key factor of success since the beginnings of EC. Deep similarities between Ensemble Learning and EC
thus appear; in both cases, diversity is used to escape from local minima, where any single "best" solution is only too easily trapped. Despite this similarity, Evolutionary Learning has most often (with some notable exceptions, see [14, 16, 18] among others) focused on single-hypothesis learning, where some single best-of-run hypothesis is returned as the solution.

However, the evolutionary population itself could be used 1**In practice, the generalization error is estimated from the**
error on a test set, disjoint from the training set. The reader is referred to [4] for a comprehensive discussion about the comparative evaluation of learning algorithms.