text: Machines learning techniques plays a preponderant role in dealing with massive amount of data and are employed in almost every possible domain. Building a high quality machine learning model to be deployed in production is a challenging task, from both, the subject matter experts and the machine learning practitioners. For a broader adoption and scalability of machine learning systems, the construction and configuration of machine learning workflow need to gain in automation. In the last few years, several techniques have been developed in this direction, known as autoML. In this paper, we present a two-stage optimization process to build data pipelines and configure machine learning algorithms. First, we study the impact of data pipelines compared to algorithm configuration in order to show the importance of data preprocessing over hyperparameter tuning. The second part presents policies to efficiently allocate search time between data pipeline construction and algorithm configuration. Those policies are agnostic from the metaoptimizer. Last, we present a metric to determine if a data pipeline is specific or independent from the algorithm, enabling fine-grain pipeline pruning and meta-learning for the coldstart problem.
  @cite_28: Many well known methods for seeking the extremum had been developed on the basis of quadratic approximation.
  @cite_16: Bayesian optimization is an approach to optimizing objective functions that take a long time (minutes or hours) to evaluate. It is best-suited for optimization over continuous domains of less than 20 dimensions, and tolerates stochastic noise in function evaluations. It builds a surrogate for the objective and quantifies the uncertainty in that surrogate using a Bayesian machine learning technique, Gaussian process regression, and then uses an acquisition function defined from this surrogate to decide where to sample. In this tutorial, we describe how Bayesian optimization works, including Gaussian process regression and three common acquisition functions: expected improvement, entropy search, and knowledge gradient. We then discuss more advanced techniques, including running multiple function evaluations in parallel, multi-fidelity and multi-information source optimization, expensive-to-evaluate constraints, random environmental conditions, multi-task Bayesian optimization, and the inclusion of derivative information. We conclude with a discussion of Bayesian optimization software and future research directions in the field. Within our tutorial material we provide a generalization of expected improvement to noisy evaluations, beyond the noise-free setting where it is more commonly applied. This generalization is justified by a formal decision-theoretic argument, standing in contrast to previous ad hoc modifications.
  @cite_32: Bayesian optimization is a sample-efficient approach for global optimization and relies on acquisition functions to guide the search process. Maximizing these functions is inherently complicated, especially in the parallel setting, where acquisition functions are routinely non-convex, high-dimensional and intractable. We present two modern approaches for maximizing acquisition functions and show that 1) sample-path derivatives can be used to optimize acquisition functions and 2) parallel formulations of many acquisition functions are submodular and can therefore be efficiently maximized in greedy fashion with guaranteed near-optimality.
summary: An acquisition function is used to determined the next configuration to be sampled. Most of those functions are based on Bayesian optimization @cite_32 @cite_16 . One popular strategy is to select @math such that it maximizes the expected improvement @cite_28 .