

![0_image_0.png](0_image_0.png)

## 7.3 Results For The Sampling Algorithm

The budget-sensitive progressive sampling algorithm was applied to the phone, adult, covertype, kr-vs-kp, weather, letter-a and blackjack data sets using both error rate and AUC to measure classifier performance. However, the method for setting the beam (described in lines 6-12 in Table 7) 
was modified so that the results from the experiments described in Section 7.1 (and detailed in Appendix B), which evaluate only the 13 listed class distributions, could be used. Specifically, at each iteration the low end (high end) of the beam is set to the class distribution specified in Appendix B that is just below (above) the best performing class distribution from the prior iteration. For example, if in one iteration the best performing class distribution contains 30% minority-class examples, then in the next iteration the bottom of the beam is set to include 20% 
minority-class examples and the top of the beam to include 40% minority-class examples (of the 13 sampled class distributions, these are the closest to the 30% class distribution). Although this will sometimes allow the beam to span a range greater than µ (2), in practice this does not result in a problem—for the seven data sets all examples requested by the algorithm are included in the final training set. In addition, a slight improvement was made to the algorithm. Specifically, for any iteration, if the number of examples already in hand (procured in previous iterations) is sufficient to evaluate additional class distributions, then the beam is widened to include these additional class distributions (this can happen because during the first iteration the beam is set very wide). 

The performance of this sampling algorithm is summarized in Table 8, along with the performance of two other strategies for selecting the class distribution of the training data. The first of the two additional strategies, the "Pick Natural/Balanced Strategy," is based on the guidelines suggested by the empirical results from Section 6. This strategy selects the natural distribution when error rate is the performance metric and the balanced class distribution when AUC is the 