

![0_image_0.png](0_image_0.png)

Finally, to assess whether pruning would have improved performance, consider the second column in Table 5, which displays the error rates that result from using C4.5 with pruning on the natural distribution (recall from Section 4.2 that this is the only case when C4.5's pruning strategy will give unbiased results). A "+"/"x" in the second column indicates that C4.5 with pruning outperforms/underperforms C4.5 without pruning, when learning from the natural distribution. 

Note that C4.5 with pruning *underperforms* C4.5 without pruning for 17 of the 26 data sets, which leads us to conclude that C4.5 without pruning is a reasonable learner. Furthermore, in no case does C4.5 with pruning generate a classifier within the optimal range when C4.5 without pruning does not also generate a classifier within this range. 

## 6.3 The Relationship Between Class Distribution And Auc

The performance of the induced classifiers, using AUC as the performance measure, is displayed in Table 6. When viewing these results, recall that for AUC *larger* values indicate improved performance. The relative improvement in classifier performance is again specified in the last two columns, but now the relative improvement in performance is calculated in terms of the area above the ROC curve (i.e., 1 - AUC). We use the area above the ROC curve because it better reflects the relative improvement—just as in Table 5 relative improvement is specified in terms of the change in error rate instead of the change in accuracy. As before, the relative improvements are shown in bold only if we are more than 90% confident that they reflect a true improvement in performance (i.e., t-test value ≤ .10). 

In general, the optimum ranges appear to be centered near, but slightly to the right, of the balanced class distribution. For 12 of the 26 data sets the optimum range does not include the natural distribution (i.e., the third column is not shaded). Note that for these data sets, with the exception of the solar-flare data set, the class distributions within the optimal range contain more minority-class examples than the natural class distribution. Based on these results we conclude even more strongly for AUC (i.e., for cost-sensitive classification and for ranking) than for accu-