Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Implicit Gradient Regularization   Gradient descent can be surprisingly good at optimizing deep neural networks
without overfitting and without explicit regularization. We find that the
discrete steps of gradient descent implicitly regularize models by penalizing
gradient descent trajectories that have large loss gradients. We call this
Implicit Gradient Regularization (IGR) and we use backward error analysis to
calculate the size of this regularization. We confirm empirically that implicit
gradient regularization biases gradient descent toward flat minima, where test
errors are small and solutions are robust to noisy parameter perturbations.
Furthermore, we demonstrate that the implicit gradient regularization term can
be used as an explicit regularizer, allowing us to control this gradient
regularization directly. More broadly, our work indicates that backward error
analysis is a useful theoretical approach to the perennial question of how
learning rate, model size, and parameter regularization interact to determine
the properties of overparameterized models optimized with gradient descent.

21
The categories of this paper are: adap-org
9
The categories of this paper are: nlin
26
The categories of this paper are: bayes-an
3
The categories of this paper are: physics
37
The categories of this paper are: supr-con
28
The categories of this paper are: alg-geom
19
The categories of this paper are: eess
23
The categories of this paper are: solv-int
4
The categories of this paper are: cond-mat
2
The categories of this paper are: math
29
The categories of this paper are: funct-an
34
The categories of this paper are: chem-ph
32
The categories of this paper are: atom-ph
18
The categories of this paper are: econ
10
The categories of this paper are: q-bio
24
The categories of this paper are: dg-ga