Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: An Adaptive Remote Stochastic Gradient Method for Training Neural
  Networks   We present the remote stochastic gradient (RSG) method, which computes the
gradients at configurable remote observation points, in order to improve the
convergence rate and suppress gradient noise at the same time for different
curvatures. RSG is further combined with adaptive methods to construct ARSG for
acceleration. The method is efficient in computation and memory, and is
straightforward to implement. We analyze the convergence properties by modeling
the training process as a dynamic system, which provides a guideline to select
the configurable observation factor without grid search. ARSG yields
$O(1/\sqrt{T})$ convergence rate in non-convex settings, that can be further
improved to $O(\log(T)/T)$ in strongly convex settings. Numerical experiments
demonstrate that ARSG achieves both faster convergence and better
generalization, compared with popular adaptive methods, such as ADAM, NADAM,
AMSGRAD, and RANGER for the tested problems. In particular, for training
ResNet-50 on ImageNet, ARSG outperforms ADAM in convergence speed and meanwhile
it surpasses SGD in generalization.

7
The categories of this paper are: hep-th
13
The categories of this paper are: quant-ph
24
The categories of this paper are: dg-ga
18
The categories of this paper are: econ
35
The categories of this paper are: mtrl-th
6
The categories of this paper are: astro-ph
36
The categories of this paper are: cmp-lg
27
The categories of this paper are: comp-gas
17
The categories of this paper are: q-fin
10
The categories of this paper are: q-bio
21
The categories of this paper are: adap-org
22
The categories of this paper are: chao-dyn
4
The categories of this paper are: cond-mat
12
The categories of this paper are: nucl-th
3
The categories of this paper are: physics
19
The categories of this paper are: eess