Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: On the Distributional Properties of Adaptive Gradients   Adaptive gradient methods have achieved remarkable success in training deep
neural networks on a wide variety of tasks. However, not much is known about
the mathematical and statistical properties of this family of methods. This
work aims at providing a series of theoretical analyses of its statistical
properties justified by experiments. In particular, we show that when the
underlying gradient obeys a normal distribution, the variance of the magnitude
of the \textit{update} is an increasing and bounded function of time and does
not diverge. This work suggests that the divergence of variance is not the
cause of the need for warm up of the Adam optimizer, contrary to what is
believed in the current literature.

19
The categories of this paper are: eess
22
The categories of this paper are: chao-dyn
24
The categories of this paper are: dg-ga
26
The categories of this paper are: bayes-an
14
The categories of this paper are: math-ph
25
The categories of this paper are: patt-sol
32
The categories of this paper are: atom-ph
15
The categories of this paper are: nucl-ex
34
The categories of this paper are: chem-ph
13
The categories of this paper are: quant-ph
31
The categories of this paper are: ao-sci
4
The categories of this paper are: cond-mat
23
The categories of this paper are: solv-int
10
The categories of this paper are: q-bio
36
The categories of this paper are: cmp-lg
8
The categories of this paper are: hep-ex