Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: On the Convergence of Adaptive Gradient Methods for Nonconvex
  Optimization   Adaptive gradient methods are workhorses in deep learning. However, the
convergence guarantees of adaptive gradient methods for nonconvex optimization
have not been thoroughly studied. In this paper, we provide a fine-grained
convergence analysis for a general class of adaptive gradient methods including
AMSGrad, RMSProp and AdaGrad. For smooth nonconvex functions, we prove that
adaptive gradient methods in expectation converge to a first-order stationary
point. Our convergence rate is better than existing results for adaptive
gradient methods in terms of dimension, and is strictly faster than stochastic
gradient decent (SGD) when the stochastic gradients are sparse. To the best of
our knowledge, this is the first result showing the advantage of adaptive
gradient methods over SGD in nonconvex setting. In addition, we also prove high
probability bounds on the convergence rates of AMSGrad, RMSProp as well as
AdaGrad, which have not been established before. Our analyses shed light on
better understanding the mechanism behind adaptive gradient methods in
optimizing nonconvex objectives.

36
The categories of this paper are: cmp-lg
3
The categories of this paper are: physics
8
The categories of this paper are: hep-ex
13
The categories of this paper are: quant-ph
20
The categories of this paper are: acc-phys
37
The categories of this paper are: supr-con
15
The categories of this paper are: nucl-ex
32
The categories of this paper are: atom-ph
23
The categories of this paper are: solv-int
22
The categories of this paper are: chao-dyn
25
The categories of this paper are: patt-sol
11
The categories of this paper are: hep-lat
9
The categories of this paper are: nlin
34
The categories of this paper are: chem-ph
17
The categories of this paper are: q-fin
24
The categories of this paper are: dg-ga