number). This makes it possible, without replicating any examples, to generate any class distribution for training-set size S. Each training set is then formed by random sampling from the remaining data, without replacement, such that the desired class distribution is achieved. For the experiments described in this article, the class distribution of the training set is varied so that the minority class accounts for between 2% and 95% of the training data. 

## 4.2 C4.5 And Pruning

The experiments in this article use C4.5, a program for inducing classification trees from labeled examples (Quinlan, 1993). C4.5 uses the uncorrected frequency-based estimate to label the leaves of the decision tree, since it assumes that the training data approximate the true, underlying distribution. Given that we modify the class distribution of the training set, it is essential that we use the corrected estimates to re-label the leaves of the induced tree. The results presented in the body of this article are based on the use of the corrected versions of the frequency-based and Laplace estimates (described in Table 1), using a probability threshold of .5 to label the leaves of the induced decision trees. 

C4.5 does not factor in differences between the class distributions of the training and test sets—we adjust for this as a post-processing step. If C4.5's pruning strategy, which attempts to minimize error rate, were allowed to execute, it would prune based on a false assumption (viz., 
that the test distribution matches the training distribution). Since this may negatively affect the generated classifier, except where otherwise indicated all results are based on C4.5 without pruning. This decision is supported by recent research, which indicates that when target misclassification costs (or class distributions) are unknown then standard pruning does not improve, and may degrade, generalization performance (Provost & Domingos, 2001; Zadrozny & Elkan, 2001; Bradford et al., 1998; Bauer & Kohavi, 1999). Indeed, Bradford et al. (1998) found that even if the pruning strategy is adapted to take misclassification costs and class distribution into account, this does not generally improve the performance of the classifier. Nonetheless, in order to justify using C4.5 without pruning, we also present the results of C4.5 with pruning when the training set uses the natural distribution. In this situation C4.5's assumption about rtrain approximating ρ is valid and hence its pruning strategy will perform properly. Looking ahead, these results show that C4.5 without pruning indeed performs competitively with C4.5 with pruning. 

## 4.3 Evaluating Classifier Performance

A variety of metrics for assessing classifier performance are based on the terms listed in the confusion matrix shown below. 

|                 | t(x)                |                     |
|-----------------|---------------------|---------------------|
|                 | Positive Prediction | Negative Prediction |
| Actual Positive | tp (true positive)  | fn (false negative) |
| Actual Negative | fp (false positive) | tn (true negative)  |
| c(x)            |                     |                     |

Table 3 summarizes eight such metrics. The metrics described in the first two rows measure the ability of a classifier to classify positive and negative *examples* correctly, while the metrics described in the last two rows measure the effectiveness of the *predictions* made by a classifier. 

For example, the positive predictive value (PPV), or *precision*, of a classifier measures the fraction of positive predictions that are correctly classified. The metrics described in the last two 