# Issues In Stacked Generalization

K MTIN G Q D EA K IN . ED U . A U
Kai Ming Ting School of Computing and Mathematics Deakin University, Australia.

## Ian H.  Witten

Department of Computer Science University of Waikato, New Zealand.

IHW © CS  WAIKATO  AC  NZ

## Abstract

Stacked generalization is a general method of using a high-level model to combine lowerlevel models to achieve greater predictive accuracy. In this paper we address two crucial issues which have been considered to be a 'black art' in classification tasks ever since the introduction of stacked generalization in 1992 by Wolpert: the type of generalizer that is suitable to derive the higher-level model, and the kind of attributes that should be used as its input. We find that best results are obtained when the higher-level model combines the confidence (and not just the predictions) of the lower-level ones.

We demonstrate the effectiveness of stacked generalization for combining three different types of learning algorithms for classification tasks. We also compare the performance of stacked generalization with majority vote and published results of arcing and bagging.

## 1. Introduction

Stacked generalization is a way of combining multiple models that have been learned for a classification task  Wolpert which has also been used for regression   Breiman a and even unsupervised learning  Smyth Wolpert Typically different learning algorithms learn different models for the task at hand and in the most common form of stacking the first step is to collect the output of each model into a new set of data  For each instance in the original training set this data set represents every model s prediction of that instance s class along with its true classification During this step care is taken to ensure that the models are formed from a batch of training data that does not include the instance in question  in just the same way as ordinary cross validation  The new data are treated as the data for another learning problem  and in the second step a learning algorithm is employed to solve this problem In Wolpert s terminology the original data and the models constructed for them in the first step are referred to as level-O data and level-O models respectively while the set of cross validated data and the second stage learning algorithm are referred to as level-1 data and the level-1 generalizer In this paper we show how to make stacked generalization work for classification tasks by addressing two crucial issues which Wolpert originally described as  black art and have not been resolved since The two issues are i the type of attributes that should be used to form level   data  and  ii  the type of level   generalizer in order to get improved accuracy using the stacked generalization method a  demonstrated the success of stacked generalization in the setting of Breiman ordinary regression  The level  models are regression trees of different sizes or linear
©1999 AI Access Foundation and Morgan Kaufmann Publishers. All rights reserved.