Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Directional Analysis of Stochastic Gradient Descent via von Mises-Fisher
  Distributions in Deep learning   Although stochastic gradient descent (SGD) is a driving force behind the
recent success of deep learning, our understanding of its dynamics in a
high-dimensional parameter space is limited. In recent years, some researchers
have used the stochasticity of minibatch gradients, or the signal-to-noise
ratio, to better characterize the learning dynamics of SGD. Inspired from these
work, we here analyze SGD from a geometrical perspective by inspecting the
stochasticity of the norms and directions of minibatch gradients. We propose a
model of the directional concentration for minibatch gradients through von
Mises-Fisher (VMF) distribution, and show that the directional uniformity of
minibatch gradients increases over the course of SGD. We empirically verify our
result using deep convolutional networks and observe a higher correlation
between the gradient stochasticity and the proposed directional uniformity than
that against the gradient norm stochasticity, suggesting that the directional
statistics of minibatch gradients is a major factor behind SGD.

2
The categories of this paper are: math
33
The categories of this paper are: plasm-ph
19
The categories of this paper are: eess
15
The categories of this paper are: nucl-ex
28
The categories of this paper are: alg-geom
17
The categories of this paper are: q-fin
7
The categories of this paper are: hep-th
11
The categories of this paper are: hep-lat
20
The categories of this paper are: acc-phys
0
The categories of this paper are: hep-ph
12
The categories of this paper are: nucl-th
30
The categories of this paper are: q-alg
31
The categories of this paper are: ao-sci
13
The categories of this paper are: quant-ph
3
The categories of this paper are: physics
10
The categories of this paper are: q-bio