Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Budgeted Training: Rethinking Deep Neural Network Training Under
  Resource Constraints   In most practical settings and theoretical analyses, one assumes that a model
can be trained until convergence. However, the growing complexity of machine
learning datasets and models may violate such assumptions. Indeed, current
approaches for hyper-parameter tuning and neural architecture search tend to be
limited by practical resource constraints. Therefore, we introduce a formal
setting for studying training under the non-asymptotic, resource-constrained
regime, i.e., budgeted training. We analyze the following problem: "given a
dataset, algorithm, and fixed resource budget, what is the best achievable
performance?" We focus on the number of optimization iterations as the
representative resource. Under such a setting, we show that it is critical to
adjust the learning rate schedule according to the given budget. Among
budget-aware learning schedules, we find simple linear decay to be both robust
and high-performing. We support our claim through extensive experiments with
state-of-the-art models on ImageNet (image classification), Kinetics (video
classification), MS COCO (object detection and instance segmentation), and
Cityscapes (semantic segmentation). We also analyze our results and find that
the key to a good schedule is budgeted convergence, a phenomenon whereby the
gradient vanishes at the end of each allowed budget. We also revisit existing
approaches for fast convergence and show that budget-aware learning schedules
readily outperform such approaches under (the practical but under-explored)
budgeted training setting.

32
The categories of this paper are: atom-ph
18
The categories of this paper are: econ
19
The categories of this paper are: eess
2
The categories of this paper are: math
8
The categories of this paper are: hep-ex
30
The categories of this paper are: q-alg
23
The categories of this paper are: solv-int
28
The categories of this paper are: alg-geom
16
The categories of this paper are: stat
29
The categories of this paper are: funct-an
37
The categories of this paper are: supr-con
5
The categories of this paper are: gr-qc
24
The categories of this paper are: dg-ga
4
The categories of this paper are: cond-mat
0
The categories of this paper are: hep-ph
7
The categories of this paper are: hep-th