

![0_image_0.png](0_image_0.png)

on the test dataset The result is expressed as the average error rate of the W fold cross validation Note that this cross validation is used for evaluation of the entire procedure whereas the J fold cross validation mentioned in Section is the internal operation of sssseens to the start of in the experiments In this section  we present results of model combination using level   models M and M ' as well as a model selection method  employing the same J fold cross validation pro cdure  Note that the only difference between model combination and model selection here is whether the level   learning is employed or not Table shows the average error rates obtained using W fold cross validation of C
NB and IB and BestCV which is the best of the three selected using J fold cross validation  As expected  Best CV is almost always the classifier with the lowest error rate 3 Table shows the result of stacked generalization using the level   model M  for which data comprise the classifications generated by the level   models and M ′  for the level which the level    data comprise the probabilities generated by the level    models  Results are shown for all four level   generalizers in each case  along with BestCV  The lowest error rate for each dataset is given in bold Table summarizes the results in Table   in terms of a comparison of ea level model with BestCV totaled over all datasets  Clearly  the best level   model is M ′ derived using MLR. It performs better than BestCV in nine datasets and equally well in the tenth The best performing M is derived from NB  which performs better than BestCV in seven datasets but significantly worse in two  Waveform and Vowel   We regard a difference of more than two standard errors as significant confidence The standard error figures are omitted in this table to increase readability The datasets are shown in the order of increasing size  MLR performs significantly better than BestCV in the four largest datasets This indicates that stacked generalization is more likely to give significant improvements in predictive accuracy if the volume of data is large a direct consequence of more accurate estimation using cross validation