Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Accumulated Gradient Normalization   This work addresses the instability in asynchronous data parallel
optimization. It does so by introducing a novel distributed optimizer which is
able to efficiently optimize a centralized model under communication
constraints. The optimizer achieves this by pushing a normalized sequence of
first-order gradients to a parameter server. This implies that the magnitude of
a worker delta is smaller compared to an accumulated gradient, and provides a
better direction towards a minimum compared to first-order gradients, which in
turn also forces possible implicit momentum fluctuations to be more aligned
since we make the assumption that all workers contribute towards a single
minima. As a result, our approach mitigates the parameter staleness problem
more effectively since staleness in asynchrony induces (implicit) momentum, and
achieves a better convergence rate compared to other optimizers such as
asynchronous EASGD and DynSGD, which we show empirically.

34
The categories of this paper are: chem-ph
29
The categories of this paper are: funct-an
24
The categories of this paper are: dg-ga
28
The categories of this paper are: alg-geom
32
The categories of this paper are: atom-ph
26
The categories of this paper are: bayes-an
19
The categories of this paper are: eess
5
The categories of this paper are: gr-qc
14
The categories of this paper are: math-ph
23
The categories of this paper are: solv-int
11
The categories of this paper are: hep-lat
35
The categories of this paper are: mtrl-th
22
The categories of this paper are: chao-dyn
13
The categories of this paper are: quant-ph
17
The categories of this paper are: q-fin
2
The categories of this paper are: math