as a pool for recruiting the elements of an ensemble, enabling "Ensemble Learning for Free". Previous work along this line will be described in Section 2, mostly based on using an evolutionary algorithm as weak learner [17], or using evolutionary diversity-enforcing heuristics [16, 18].

In this paper, the "Evolutionary Ensemble Learning For Free" claim is empirically examined along two directions.

The first direction is that of the classifier diversity; a new learning-oriented fitness function is proposed, inspired by the co-evolution framework [13] and generalizing the diversity-enforcing fitness proposed by [18]. The second direction is that of the selection of the ensemble classifiers within the evolutionary population(s). Selecting the best classifiers in a pool amounts to a feature selection problem, that is, a combinatorial optimization problem [12]. A greedy set-covering approach is used, build on a margin-based criterion inspired by Schapire et al. **[23]. Finally, the paper presents two** Evolutionary Ensemble Learning (EEL) approaches, called Off-EEL and On**-EEL, respectively tackling the selection of**
the ensemble classifiers in the final population, or along evolution.

Paper structure is as follows. Section 2 reviews and discusses some work relevant to Evolutionary Ensemble Learning. Section 3 describes the two proposed approaches OffEEL and On**-EEL, introducing the specific fitness function**
and the ensemble classifier selection procedure. Experimental results based on benchmark problems from the UCI
repository are reported in Section 4. The paper concludes with some perspectives for further research, discussing the priorities for a tight coupling of Ensemble Learning with Evolutionary Optimization in terms of dynamic systems [22].

## 2. Related Work

Interestingly, some early approaches in Evolutionary Learning were rooted on Ensemble Learning ideas2**. The Michigan**
approach [14] evolves a population made of rules, whereas the Pittsburgh approach evolves a population made of sets of rules. What is gained in flexibility and tractability in the Michigan approach is compensated by the difficulty of assessing a single rule, for the following reason. A rule usually only covers a part of the example space; gathering the best rules (e.g. the rules with highest accuracy) does not result in the best ruleset. Designing an efficient fitness function, such that a good quality ruleset could be extracted from the final population, was found a tricky task.

In the last decade, Ensemble Learning has been explored within Evolutionary Learning, chiefly in the context of Genetic Programming (GP). A first trend directly inspired from Bagging and Boosting aims at reducing the fitness computation cost [7, 16] and/or dealing with datasets which do not fit in memory [24]. For instance, Iba [16] divided the GP
population into several sub-populations which are evaluated on subsets of the training set. Folino et al. **[7] likewise sampled the training set in a Bagging-like mode in the context**
of parallel cellular GP. Song et al. **[24] used Boosting-like**
heuristics to deal with training sets that do not fit in memory; the training set is divided into folds, one of which is loaded in memory and periodically replaced; at each generation, small subsets are selected from the current fold to 2**Learning Classifier Systems (LCS, [14, 15]) are mostly devoted to Reinforcement Learning, as opposed to Supervised**
Machine Learning; therefore they will not be considered in the paper.

compute the fitness function, where the selection is nicely based on a mixture of uniform and Boosting-like distributions.

The use of Evolutionary Algorithms as weak learners within a standard Bagging or Boosting approach has also been investigated. Boosting approaches for GP have been applied for instance to classification [21] or symbolic regression [17]:
each run delivers a GP tree minimizing the weighted sum of the training errors, and the weights were computed as in standard Boosting [8]. While such ensembles of GP trees result, as expected, in a much lower variance of the performance, they do not fully exploit the population-based nature of GP, as independent runs are launched to learn successive classifiers.

Liu et al. **[18] proposed a tight coupling between Evolutionary Algorithms and Ensemble Learning. They constructed an ensemble of Neural Networks, using a modified**
back-propagation algorithm to enforce the diversity of the networks; specifically, the back-propagation aims at both minimizing the training error and maximizing the negative correlation of the current network with respect to the current population. Further, the fitness associated to each network is the sum of the weights of all examples it correctly classifies, where the weight of each example is inversely proportional to the number of classifiers that correctly classify this example. While this approach nicely suggests that ensemble learning is a Multiple Objective Optimization (MOO) problem (minimize the error rate and maximize the diversity),
it classically handles the MOO problem as a fixed weighted sum of the objectives.

The MOO perspective was further investigated by Chandra and Yao in the DIVACE system, a highly sophisticated system for the multi-level evolution of ensemble of classifiers [2, 3]. In [3], the top-level evolution simultaneously minimizes the error rate (accuracy) and maximizes the negative correlation (diversity). In [2], the negative correlationinspired criterion is replaced by a **pairwise failure crediting**;
the difference concerns the misclassification of examples that are correctly classified by other classifiers. Finally, the ensemble is constructed either by keeping all classifiers in the final population, or by clustering the final population (after their phenotypic distance) and selecting a classifier in each cluster.

While the MOO perspective nicely captures the interplay of the accuracy and diversity goals within Ensemble Learning, the selection of the classifiers in the genetic pool as done in [2, 3] does not fully exploit the possibilities of evolutionary optimization, in two respects. On the one hand, it only considers the final population that usually involves up to a few hundred classifiers, while learning ensembles commonly involve some thousand classifiers. On the other hand, clustering-based selection proceeds on the basis of the phenotypic distance between classifiers, considering again that all examples are equally important, while the higher stress put on harder examples is considered the source of the better Boosting efficiency [5].

## 3. Ensemble Learning For Free

After the above discussion, Evolutionary Ensemble Learning (EEL) involves two critical issues: i) how to enforce both the predictive accuracy and the diversity of the classifiers in the population, and across generations; ii) how to best select the ensemble classifiers, from either the final population