Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Understanding and Scheduling Weight Decay   Weight decay is a popular and even necessary regularization technique for
training deep neural networks that generalize well. Previous work usually
interpreted weight decay as a Gaussian prior from the Bayesian perspective.
However, weight decay sometimes shows mysterious behaviors beyond the
conventional understanding. For example, the optimal weight decay value tends
to be zero given long enough training time. Moreover, existing work typically
failed to recognize the importance of scheduling weight decay during training.
Our work aims at theoretically understanding novel behaviors of weight decay
and designing schedulers for weight decay in deep learning. This paper mainly
has three contributions. First, we propose a novel theoretical interpretation
of weight decay from the perspective of learning dynamics. Second, we propose a
novel weight-decay linear scaling rule for large-batch training that
proportionally increases weight decay rather than the learning rate as the
batch size increases. Third, we provide an effective learning-rate-aware
scheduler for weight decay, called the Stable Weight Decay (SWD) method, which,
to the best of our knowledge, is the first practical design for weight decay
scheduling. In our various experiments, the SWD method often makes improvements
over $L_{2}$ Regularization and Decoupled Weight Decay.

7
The categories of this paper are: hep-th
14
The categories of this paper are: math-ph
17
The categories of this paper are: q-fin
30
The categories of this paper are: q-alg
24
The categories of this paper are: dg-ga
34
The categories of this paper are: chem-ph
23
The categories of this paper are: solv-int
19
The categories of this paper are: eess
15
The categories of this paper are: nucl-ex
18
The categories of this paper are: econ
0
The categories of this paper are: hep-ph
6
The categories of this paper are: astro-ph
25
The categories of this paper are: patt-sol
11
The categories of this paper are: hep-lat
27
The categories of this paper are: comp-gas
4
The categories of this paper are: cond-mat