shows that a small performance variation among member models is a necessary condition for majority vote as employed by bagging to work well It is worth noting that arcing and bagging can be incorporated into the framework of stacked generalization by using arced or bagged models as level   models  Ting Witten show one possible way of incorporating bagged models with level   learning  em ploying MLR instead of voting In this implementation L is used as a test set for each of the bagged models to derive level   data rather than the cross validated data  This is viable because each bootstrap sample leaves out about         of the examples   Ting Witten show that bag stacking almost always has higher predictive accuracy than bagging models derived from either C    or NB  Note that the only difference here is whether an adaptive level model or a simple majority vote is employed c arcing and bagging can only improve the predic According to Breiman b tive accuracy of learning algorithms that are unstable 5 An unstable learning algorithm is one for which small perturbations in the training set can produce large changes in the derived model  Decision trees and neural networks are unstable NB and IB  are stable Stacking works with both While MLR is the most successful candidate for level   learning that we have found other algorithms might work equally well  One candidate is neural networks  However we have experimented with back propagation neural networks for this purpose and found that they have a much slower learning rate than MLR. For example MLR only took seconds for the neural network in the nettalk dataset  while s seconds as compare to both have the same error rate Other possible candidates are the multinomial logit model acobs which is a special case of generalized linear models  McCullagh ordan and the supra Bayesian procedure    acobs Nelder which treats the level models  confidence as data that may be combined with prior distribution of level    models via Bayes  rule

## 5. Related Work

1. Our analysis of stacked generalization was motivated by that of Breiman a discussed earlier  and LeBlanc     Tibshirani LeBlanc    Tibshirani examine the stacking of a linear discriminant and a nearest neighbor classifier and show that  for one artificial dataset a method similar to MLR performs better with non negativity constraints than without  Our results in Section    show that these constraints are irrelevant to MLRs predictive accuracy in the classification situation LeBlanc Tibshirani and  Ting Witten use a version of MLR that employs all class probabilities from each level   model to induce each linear regression  In this case  the linear regression for class   is

$$L\,R_{\ell}\,\,x$$
$$\sum_{k}^{K}\sum_{i}^{I}\alpha_{k\,i\,\ell}P_{k\,i}\,\,\,x\,\,\,.$$

This implementation requires the fitting of KI parameters as compared to K parameters for the version used in this paper  see the corresponding formula in Section Both