tion D. Therefore, the training data are less likely to include (enough) instances of all of the minority-class subconcepts in the concept space, and the learner may not have the opportunity to represent all truly positive regions in LP. Because of this, some minority-class test examples will be mistakenly classified as belonging to the majority class. 

Finally, it is worth noting that PPV > NPV does not imply that FN > FP. That is, having more error-prone minority predictions does not imply that the minority-class examples will be misclassified more often than majority-class examples. Indeed, a higher error rate for minority predictions means more majority-class test examples will be misclassified. The reason we generally observe a lower error rate for the majority-class test examples (FN > FP) is because the majority class is predicted far more often than the minority class. 

## 6. The Effect Of Training-Set Class Distribution On Classifier Performance

We now turn to the central questions of our study: how do different training-set class distributions affect the performance of the induced classifiers and which class distributions lead to the best classifiers? We begin by describing the methodology for determining which class distribution performs best. Then, in the next two sections, we evaluate and analyze classifier performance for the twenty-six data sets using a variety of class distributions. We use error rate as the performance metric in Section 6.2 and AUC as the performance metric in Section 6.3. 

## 6.1 Methodology For Determining The Optimum Training Class Distribution(S)

In order to evaluate the effect of class distribution on classifier performance, we vary the training-set class distributions for the twenty-six data sets using the methodology described in Section 4.1. We evaluate the following twelve class distributions (expressed as the percentage of minority-class examples): 2%, 5%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, and 95%. For each data set we also evaluate the performance using the naturally occurring class distribution. 

Before we try to determine the "best" class distribution for a training set, there are several issues that must be addressed. First, because we do not evaluate every possible class distribution, we can only determine the best distribution among the 13 evaluated distributions. Beyond this concern, however, is the issue of statistical significance and, because we generate classifiers for 13 training distributions, the issue of multiple comparisons (Jensen & Cohen, 2000). Because of these issues we cannot always conclude that the distribution that yields the best performing classifiers is truly the best one for training. We take several steps to address the issues of statistical significance and multiple comparisons. To enhance our ability to identify true differences in classifier performance with respect to changes in class distribution, all results presented in this section are based on 30 runs, rather than the 10 runs employed in Section 5. Also, rather than trying to determine the best class distribution, we adopt a more conservative approach, and instead identify an "optimal range" of class distributions—a range in which we are confident the best distribution lies. To identify the optimal range of class distributions, we begin by identifying, for each data set, the class distribution that yields the classifiers that perform best over the 30 runs. We then perform t-tests to compare the performance of these 30 classifiers with the 30 classifiers generated using each of the other twelve class distributions (i.e., 12 t-tests each with n=30 data points). If a t-test yields a probability ≤ .10 then we conclude that the "best" distribution is different from the "other" distribution 
(i.e., we are at least 90% confident of this); otherwise we cannot conclude that the class distributions truly perform differently and therefore "group" the distributions together. These grouped 