
## 5. Learning From Unbalanced Data Sets

We now analyze the classifiers induced from the twenty-six naturally unbalanced data sets described in Table 2, focusing on the differences in performance for the minority and majority classes. We do not alter the class distribution of the training data in this section, so the classifiers need not be adjusted using the method described in Section 3. However, so that these experiments are consistent with those in Section 6 that use the natural distribution, the size of the training set is reduced, as described in Section 4.1. 

Before addressing these differences, it is important to discuss an issue that may lead to confusion if left untreated. Practitioners have noted that learning performance often is unsatisfactory when learning from data sets where the minority class is substantially underrepresented. In particular, they observe that there is a large error rate for the minority class. As should be clear from Table 3 and the associated discussion, there are two different notions of "error rate for the minority class": the minority-class predictions could have a high error rate (large PPV ) or the minority-class test examples could have a high error rate (large FN). When practitioners observe that the error rate is unsatisfactory for the minority class, they are usually referring to the fact that the minority-class *examples* have a high error rate (large FN). The analysis in this section will show that the error rate associated with the minority-class predictions ( PPV ) and the minority-class test examples (FN) both are much larger than their majority-class counterparts ( NPV and FP, respectively). We discuss several explanations for these observed differences. 

## 5.1 Experimental Results

The performances of the classifiers induced from the twenty-six unbalanced data sets are described in Table 4. This table warrants some explanation. The first column specifies the data set name while the second column, which for convenience has been copied from Table 2, specifies the percentage of minority-class examples in the natural class distribution. The third column specifies the percentage of the total test errors that can be attributed to the test examples that belong to the minority class. By comparing the values in columns two and three we see that in all cases a disproportionately large percentage of the errors come from the minority-class examples. For instance, minority-class examples make up only 3.9% of the letter-a data set but contribute 58.3% of the errors. Furthermore, for 22 of 26 data sets a *majority* of the errors can be attributed to minority-class examples. 

The fourth column specifies the number of leaves labeled with the minority and majority classes and shows that in all but two cases there are fewer leaves labeled with the minority class than with the majority class. The fifth column, "Coverage," specifies the average number of training examples that each minority-labeled or majority-labeled leaf classifies ("covers"). These results indicate that the leaves labeled with the minority class are formed from far fewer training examples than those labeled with the majority class. 

The "Prediction ER" column specifies the error rates associated with the minority-class and majority-class predictions, based on the performance of these predictions at classifying the test examples. The "Actuals ER" column specifies the classification error rates for the minority and majority class examples, again based on the test set. These last two columns are also labeled using the terms defined in Section 2 ( PPV , NPV , FN, and FP). As an example, these columns show that for the letter-a data set the minority-labeled predictions have an error rate of 32.5% 
while the majority-labeled predictions have an error rate of only 1.7%, and that the minorityclass test examples have a classification error rate of 41.5% while the majority-class test exam-