since it accentuates examples currently classified incorrectly by previous members of the ensemble.

Addemup **(Opitz & Shavlik, 1996a, 1996b) is another example of an approach that**
directly tries to create a diverse ensemble. Addemup **uses genetic algorithms to search**
explicitly for a highly diverse set of accurate trained networks. Addemup **works by first**
creating an initial population, then uses genetic operators to create new networks continually, keeping the set of networks that are highly accurate while disagreeing with each other as much as possible. Addemup **is also effective at incorporating prior knowledge, if**
available, to improve the quality of its ensemble.

An alternate approach to the ensemble framework is to train individual networks on a subtask, and to then combine these predictions with a "gating" function that depends on the input. Jacobs et al.'s (1991) adaptive mixtures of local experts, Baxt's (1992)
method for identifying myocardial infarction, and Nowlan and Sejnowski's (1992) visual model all train networks to learn specific subtasks. The key idea of these techniques is that a decomposition of the problem into specific subtasks might lead to more efficient representations and training (Hampshire & Waibel, 1989).

Once a problem is broken into subtasks, the resulting solutions need to be combined.

Jacobs et al. (1991) propose having the gating function be a network that *learns* **how to**
allocate examples to the experts. Thus the gating network allocates each example to one or more experts, and the backpropagated errors and resulting weight changes are then restricted to these networks (and the gating function). Tresp and Taniguchi (1995) propose a method for determining the gating function *after* **the problem has been decomposed and**
the experts trained. Their gating function is an input-dependent, linear-weighting function that is determined by a combination of the networks' diversity on the current input with the likelihood that these networks have seen data "near" that input.

Although the mixtures of experts and ensemble paradigms seem very similar, they are in fact quite distinct from a statistical point of view. The mixtures-of-experts model makes the assumption that a single expert is responsible for each example. In this case, each expert is a model of a region of the input space, and the job of the gating **function is to decide from**
which model the data point originates. Since each network in **the ensemble approach learns**
the whole task rather than just some subtask and thus makes no **such mutual exclusivity**
assumption, ensembles are appropriate when no one model is highly likely to be correct for any one point in the input space.

## 6. Conclusions

This paper presents a comprehensive empirical evaluation of Bagging and Boosting for neural networks and decision trees. Our results demonstrate that a Bagging ensemble nearly always outperforms a single classifier. Our results also show that a Boosting ensemble can greatly outperform both Bagging and a single classifier. **However, for some data sets** Boosting may show zero gain or even a decrease in performance **from a single classifier.** Further tests indicate that Boosting may suffer from overfitting in the presence of noise which may explain some of the decreases in performance for Boosting. We also found that a simple ensemble approach of using neural networks that differ only in their random initial weight settings performed surprisingly well, often doing as well as the Bagging.