text: Cross-Validation (CV), and out-of-sample performance-estimation protocols in general, are often employed both for (a) selecting the optimal combination of algorithms and values of hyper-parameters (called a configuration) for producing the final predictive model, and (b) estimating the predictive performance of the final model. However, the cross-validated performance of the best configuration is optimistically biased. We present an efficient bootstrap method that corrects for the bias, called Bootstrap Bias Corrected CV (BBC-CV). BBC-CV's main idea is to bootstrap the whole process of selecting the best-performing configuration on the out-of-sample predictions of each configuration, without additional training of models. In comparison to the alternatives, namely the nested cross-validation and a method by Tibshirani and Tibshirani, BBC-CV is computationally more efficient, has smaller variance and bias, and is applicable to any metric of performance (accuracy, AUC, concordance index, mean squared error). Subsequently, we employ again the idea of bootstrapping the out-of-sample predictions to speed up the CV process. Specifically, using a bootstrap-based hypothesis test we stop training of models on new folds of statistically-significantly inferior configurations. We name the method Bootstrap Corrected with Early Dropping CV (BCED-CV) that is both efficient and provides accurate performance estimates.
  @cite_18: 
  @cite_14: With the increasing size of today's data sets, nding the right parameter configuration in model selection via cross-validation can be an extremely time-consuming task. In this paper we propose an improved cross-validation procedure which uses nonparametric testing coupled with sequential analysis to determine the best parameter set on linearly increasing subsets of the data. By eliminating underperforming candidates quickly and keeping promising candidates as long as possible, the method speeds up the computation while preserving the power of the full cross-validation. Theoretical considerations underline the statistical power of our procedure. The experimental evaluation shows that our method reduces the computation time by a factor of up to 120 compared to a full cross-validation with a negligible impact on the accuracy.
  @cite_22: A printed circuit assembly has a flexible printed circuit carried on a rigid panel which includes an open ended cavity. A flexible conductor strip of the printed circuit is wiped into the cavity by a sheet met al female terminal upon insertion of the terminal into one end of the cavity. The terminal has a box-like portion for receiving a male terminal inserted into the opposite end of the cavity, a first resilient tongue for biasing a terminal received in the box-like portion against an interior surface thereof and a pair of resilient tongues for biasingly engaging the conductor strip wiped into the cavity. The terminal also includes a transverse portion which biases the conductor strip against the panel outside of the cavity. Flat or partispherical dimples for contacting the conductor strip may be utilized and the terminal may include a ferrule portion.
summary: Krueger @cite_14 in 2015 introduced the so-called (CVST) which uses nonparametric testing together with sequential analysis in order to choose the best performing configuration on the basis of linearly increasing subsets of data. At each step, the Friedman @cite_22 or the Cochran's Q test @cite_18 (for regression and classification tasks respectively) are employed in order to detect statistically significant differences between configurations' performances. Then, the seemingly under-performing configurations are further tested through sequential analysis to determine which of them will be discharged. Finally, an early stopping criterion is employed to further speed up the CV process. The winner configuration is the one that has the best average ranking, based on performance, in the last few iterations specified in advance. The disadvantage of CVST is that it initially operates on smaller subsets, thus risking the early elimination of good models when the original dataset is already small.