Some practitioners believe that the naturally occurring marginal class distribution should be used for learning, so that new examples will be classified using a model built from the same underlying distribution. Other practitioners believe that the training set should contain an increased percentage of minority-class examples, because otherwise the induced classifier will not classify minority-class examples well. This latter viewpoint is expressed by the statement, "if the sample size is fixed, a balanced sample will usually produce more accurate predictions than an unbalanced 5%/95% split" (SAS, 2001). However, we are aware of no thorough prior empirical study of the relationship between the class distribution of the training data and classifier performance, so neither of these views has been validated and the choice of class distribution often is made arbitrarily—and with little understanding of the consequences. In this article we provide a thorough study of the relationship between class distribution and classifier performance and provide guidelines—as well as a progressive sampling algorithm—for determining a "good" class distribution to use for learning. 

There are two situations where the research described in this article is of direct practical use. 

When the training data must be limited due to the cost of learning from the data, then our results—and the guidelines we establish—can help to determine the class distribution that should be used for the training data. In this case, these guidelines determine how many examples of each class to *omit* from the training set so that the cost of learning is acceptable. The second scenario is when training examples are costly to procure so that the number of training examples must be limited. In this case the research presented in this article can be used to determine the proportion of training examples belonging to each class that should be *procured* in order to maximize classifier performance. Note that this assumes that one can select examples belonging to a specific class. This situation occurs in a variety of situations, such as when the examples belonging to each class are produced or stored separately or when the main cost is due to transforming the raw data into a form suitable for learning rather than the cost of obtaining the raw, labeled, data. 

Fraud detection (Fawcett & Provost, 1997) provides one example where training instances belonging to each class come from different sources and may be procured independently by class. 

Typically, after a bill has been paid, any transactions credited as being fraudulent are stored separately from legitimate transactions. Furthermore transactions credited to a customer as being fraudulent may in fact have been legitimate, and so these transactions must undergo a verification process before being used as training data. 

In other situations, labeled raw data can be obtained very cheaply, but it is the process of forming *usable* training examples from the raw data that is expensive. As an example, consider the *phone* data set, one of the twenty-six data sets analyzed in this article. This data set is used to learn to classify whether a phone line is associated with a business or a residential customer. The data set is constructed from low-level call-detail records that describe a phone call, where each record includes the originating and terminating phone numbers, the time the call was made, and the day of week and duration of the call. There may be hundreds or even thousands of call-detail records associated with a given phone line, all of which must be summarized into a single training example. Billions of call-detail records, covering hundreds of millions of phone lines, potentially are available for learning. Because of the effort associated with loading data from dozens of computer tapes, disk-space limitations and the enormous processing time required to summarize the raw data, it is not feasible to construct a data set using all available raw data. Consequently, the number of usable training examples must be limited. In this case this was done based on the class associated with each phone line—which is known. The phone data set was 