The error rate values that are not significantly different, statistically, from the lowest error rate (i.e., the comparison yields a t-test value > .10) are shaded. Thus, for the letter-a data set, the optimum range includes those class distributions that include between 2% and 10% minorityclass examples—which includes the natural distribution. The last two columns in Table 5 show the relative improvement in error rate achieved by using the best distribution instead of the natural and balanced distributions. When this improvement is statistically significant (i.e., is associated with a t-test value ≤ .10) then the value is displayed in bold. 

The results in Table 5 show that for 9 of the 26 data sets we are confident that the natural distribution is not within the optimal range. For most of these 9 data sets, using the best distribution rather than the natural distribution yields a remarkably large relative reduction in error rate. We feel that this is sufficient evidence to conclude that for accuracy, when the training-set size must be limited, it is not appropriate simply to assume that the natural distribution should be used. 

Inspection of the error-rate results in Table 5 also shows that the best distribution does not differ from the natural distribution in any consistent manner—sometimes it includes more minorityclass examples (e.g., optdigits, car) and sometimes fewer (e.g., connect-4, solar-flare). However, it is clear that for data sets with a substantial amount of class imbalance (the ones in the top half of the table), a balanced class distribution also is not the best class distribution for training, to minimize undifferentiated error rate. More specifically, none of the top-12 most skewed data sets have the balanced class distribution within their respective optimal ranges, and for these data sets the relative improvements over the balanced distributions are striking. 

Let us now consider the error-rate values for the remaining 17 data sets for which the t-test results do not permit us to conclude that the best observed distribution truly outperforms the natural distribution. In these cases we see that the error rate values for the 12 training-set class distributions usually form a unimodal, or nearly unimodal, distribution. This is the distribution one would expect if the accuracy of a classifier progressively degrades the further it deviates from the best distribution. This suggests that "adjacent" class distributions may indeed produce classifiers that perform differently, but that our statistical testing is not sufficiently sensitive to identify these differences. Based on this, we suspect that many of the observed improvements shown in the last column of Table 5 that are not deemed to be significant statistically are nonetheless meaningful. Figure 1 shows the behavior of the learned classifiers for the adult, phone, covertype, and letter-a data sets in a graphical form. In this figure the natural distribution is denoted by the "X" 
tick mark and the associated error rate is noted above the marker. The error rate for the best distribution is underlined and displayed below the corresponding data point (for these four data sets the best distribution happens to include 10% minority-class examples). Two of the curves are associated with data sets (adult, phone) for which we are >90% confident that the best distribution performs better than the natural distribution, while for the other two curves (covertype, letter-a) we are not. Note that all four curves are perfectly unimodal. It is also clear that near the distribution that minimizes error rate, changes to the class distribution yield only modest changes in the error rate—far more dramatic changes occur elsewhere. This is also evident for most data sets in Table 5. This is a convenient property given the common goal of minimizing error rate. 

This property would be far less evident if the correction described in Section 3 were not performed, since then classifiers induced from class distributions deviating from the naturally occurring distribution would be improperly biased. 