
## 4. Experimental Setup

In this section we describe the data sets analyzed in this article, the sampling strategy used to alter the class distribution of the training data, the classifier induction program used, and, finally, the metrics for evaluating the performance of the induced classifiers. 

## 4.1 The Data Sets And The Method For Generating The Training Data

The twenty-six data sets used throughout this article are described in Table 2. This collection includes twenty data sets from the UCI repository (Blake & Merz, 1998), five data sets, identified with a "+", from previously published work by researchers at AT&T (Cohen & Singer, 1999), and one new data set, the phone data set, generated by the authors. The data sets in Table 2 are listed in order of decreasing class imbalance, a convention used throughout this article. 

| % Minority       | Dataset   | % Minority   | Dataset        |         |          |      |
|------------------|-----------|--------------|----------------|---------|----------|------|
| Dataset          | Examples  | Size         | #              | Dataset | Examples | Size |
| letter-a*        | 3.9       | 20,000       | 14 network2    | 27.9    | 3,826    |      |
| pendigits*       | 8.3       | 13,821       | 15 yeast*      | 28.9    | 1,484    |      |
| abalone*         | 8.7       | 4,177        | 16 network1+   | 29.2    | 3,577    |      |
| sick-euthyroid   | 9.3       | 3,163        | 17 car*        | 30.0    | 1,728    |      |
| connect-4*       | 9.5       | 11,258       | 18 german      | 30.0    | 1,000    |      |
| optdigits*       | 9.9       | 5,620        | 19 breast-wisc | 34.5    | 699      |      |
| covertype*       | 14.8      | 581,102      | 20 blackjack+  | 35.6    | 15,000   |      |
| solar-flare*     | 15.7      | 1,389        | 21 weather+    | 40.1    | 5,597    |      |
| phone            | 18.2      | 652,557      | 22 bands       | 42.2    | 538      |      |
| letter-vowel*    | 19.4      | 20,000       | 23 market1+    | 43.0    | 3,181    |      |
| contraceptive*   | 22.6      | 1,473        | 24 crx         | 44.5    | 690      |      |
| adult            | 23.9      | 48,842       | 25 kr-vs-kp    | 47.8    | 3,196    |      |
| splice-junction* | 24.1      | 3,175        | 26 move+       | 49.9    | 3,029    |      |

In order to simplify the presentation and the analysis of our results, data sets with more than two classes were mapped to two-class problems. This was accomplished by designating one of the original classes, typically the least frequently occurring class, as the minority class and then mapping the remaining classes into the majority class. The data sets that originally contained more than 2 classes are identified with an asterisk (*) in Table 2. The letter-a data set was created from the letter-recognition data set by assigning the examples labeled with the letter "a" to the minority class; the letter-vowel data set was created by assigning the examples labeled with any vowel to the minority class. 

We generated training sets with different class distributions as follows. For each experimental run, first the test set is formed by randomly selecting 25% of the minority-class examples and 25% of the majority-class examples from the original data set, without replacement (the resulting test set therefore conforms to the original class distribution). The remaining data are available for training. To ensure that all experiments for a given data set have the same training-set sizeno matter what the class distribution of the training set—the training-set size, S, is made equal to the total number of minority-class examples still available for training (i.e., 75% of the original 