Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Noise and Fluctuation of Finite Learning Rate Stochastic Gradient
  Descent   In the vanishing learning rate regime, stochastic gradient descent (SGD) is
now relatively well understood. In this work, we propose to study the basic
properties of SGD and its variants in the non-vanishing learning rate regime.
The focus is on deriving exactly solvable results and discussing their
implications. The main contributions of this work are to derive the stationary
distribution for discrete-time SGD in a quadratic loss function with and
without momentum; in particular, one implication of our result is that the
fluctuation caused by discrete-time dynamics takes a distorted shape and is
dramatically larger than a continuous-time theory could predict. Examples of
applications of the proposed theory considered in this work include the
approximation error of variants of SGD, the effect of minibatch noise, the
optimal Bayesian inference, the escape rate from a sharp minimum, and the
stationary covariance of a few second-order methods including damped Newton's
method, natural gradient descent, and Adam.

37
The categories of this paper are: supr-con
35
The categories of this paper are: mtrl-th
26
The categories of this paper are: bayes-an
6
The categories of this paper are: astro-ph
7
The categories of this paper are: hep-th
23
The categories of this paper are: solv-int
29
The categories of this paper are: funct-an
18
The categories of this paper are: econ
15
The categories of this paper are: nucl-ex
5
The categories of this paper are: gr-qc
22
The categories of this paper are: chao-dyn
24
The categories of this paper are: dg-ga
27
The categories of this paper are: comp-gas
19
The categories of this paper are: eess
14
The categories of this paper are: math-ph
0
The categories of this paper are: hep-ph