Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: The effective noise of Stochastic Gradient Descent   Stochastic Gradient Descent (SGD) is the workhorse algorithm of deep learning
technology. At each step of the training phase, a mini batch of samples is
drawn from the training dataset and the weights of the neural network are
adjusted according to the performance on this specific subset of examples. The
mini-batch sampling procedure introduces a stochastic dynamics to the gradient
descent, with a non-trivial state-dependent noise. We characterize the
stochasticity of SGD and a recently-introduced variant, \emph{persistent} SGD,
in a prototypical neural network model. In the under-parametrized regime, where
the final training error is positive, the SGD dynamics reaches a stationary
state and we define an effective temperature from the fluctuation-dissipation
theorem, computed from dynamical mean-field theory. We use the effective
temperature to quantify the magnitude of the SGD noise as a function of the
problem parameters. In the over-parametrized regime, where the training error
vanishes, we measure the noise magnitude of SGD by computing the average
distance between two replicas of the system with the same initialization and
two different realizations of SGD noise. We find that the two noise measures
behave similarly as a function of the problem parameters. Moreover, we observe
that noisier algorithms lead to wider decision boundaries of the corresponding
constraint satisfaction problem.

28
The categories of this paper are: alg-geom
21
The categories of this paper are: adap-org
20
The categories of this paper are: acc-phys
34
The categories of this paper are: chem-ph
11
The categories of this paper are: hep-lat
17
The categories of this paper are: q-fin
31
The categories of this paper are: ao-sci
30
The categories of this paper are: q-alg
15
The categories of this paper are: nucl-ex
35
The categories of this paper are: mtrl-th
29
The categories of this paper are: funct-an
37
The categories of this paper are: supr-con
32
The categories of this paper are: atom-ph
19
The categories of this paper are: eess
18
The categories of this paper are: econ
23
The categories of this paper are: solv-int