Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Accounting for Variance in Machine Learning Benchmarks   Strong empirical evidence that one machine-learning algorithm A outperforms
another one B ideally calls for multiple trials optimizing the learning
pipeline over sources of variation such as data sampling, data augmentation,
parameter initialization, and hyperparameters choices. This is prohibitively
expensive, and corners are cut to reach conclusions. We model the whole
benchmarking process, revealing that variance due to data sampling, parameter
initialization and hyperparameter choice impact markedly the results. We
analyze the predominant comparison methods used today in the light of this
variance. We show a counter-intuitive result that adding more sources of
variation to an imperfect estimator approaches better the ideal estimator at a
51 times reduction in compute cost. Building on these results, we study the
error rate of detecting improvements, on five different deep-learning
tasks/architectures. This study leads us to propose recommendations for
performance comparisons.

26
The categories of this paper are: bayes-an
0
The categories of this paper are: hep-ph
33
The categories of this paper are: plasm-ph
20
The categories of this paper are: acc-phys
13
The categories of this paper are: quant-ph
34
The categories of this paper are: chem-ph
15
The categories of this paper are: nucl-ex
12
The categories of this paper are: nucl-th
14
The categories of this paper are: math-ph
35
The categories of this paper are: mtrl-th
31
The categories of this paper are: ao-sci
21
The categories of this paper are: adap-org
17
The categories of this paper are: q-fin
25
The categories of this paper are: patt-sol
7
The categories of this paper are: hep-th
24
The categories of this paper are: dg-ga