regressions using different number of variables  But instead of selecting the single model that works best as judged by for example cross validation Breiman used the different level regressors  output values for each member of the training set to form level   data  Then he used least squares linear regression  under the constraint that all regression coefficients be non negative as the level   generalizer  The non negativity constraint turned out to be crucial to guarantee that the predictive accuracy would be better than that achieved by selecting the single best predictor Here we show how stacked generalization can be made to work reliably in classification tasks We do this by using the output class probabilities generated by level models to form level data Then for the level generalizer we use a version of least squares linear regression adapted for classification tasks We find the use of class probabilities to be crucial for the successful application of stacked generalization in classification tasks  However the non negativity constraints found necessary by Breiman in regression are found to be irrelevant to improved predictive accuracy in our classification situation In Section we formally introduce the technique of stacked generalization and describe pertinent details of each learning algorithm used in our experiments  Section describes the results of stacking three different types of learning algorithms  Section compares stacked generalization with arcing and bagging  two recent methods that employ sampling techniques to modify the data distribution in order to produce multiple models from a single learning algorithm The following section describes related work  and the paper ends with a summary of our conclusions

## 2. Stacked Generalization

Given a data set L     { Yn, Xn , n representing the attribute values of the nth instance  randomly split the data into J almost equal parts L 1 , . . , L 1 Define L 1 and L (- i )    L - L 1 to be the test and training sets for the j th fold of a J fold cross validation  Given K learning algorithms  which we call level-0 generalizers  invoke the k th algorithm on the data in the training set L (- j ) to induce a model M (- )) for k
, . . , K  These are called level 0 models For each instance x n in C i the test set for the j th cross validation fold  let z kn denote the prediction of the model M (- 3 ) on x n At the end of the entire cross validation process the data set assembled from the outputs of the K models is

LCV   { Yn; Zln; ... ; ZKn ; n

## P N  N  N  N  N

These are the level-1 data Use some learning algorithm that we call the level-1 generalizer to derive from these data a model M for y as a function of  z 1 , . . , z K   This is the level-1 model Figure illustrates the cross validation process To complete the training process the final level models M k   k Now let us consider the classification process  which uses the models Mk k port, K
in conjunction with M  Given a new instance  models Mk produce a vector  z 1 , . . , z K
This vector is input to the level —model M—whose output is the final classification result for that instance  This completes the stacked generalization method as proposed by Wolpert and also used by Breiman Tibshirani a and LeBlanc