Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Faster Distributed Deep Net Training: Computation and Communication
  Decoupled Stochastic Gradient Descent   With the increase in the amount of data and the expansion of model scale,
distributed parallel training becomes an important and successful technique to
address the optimization challenges. Nevertheless, although distributed
stochastic gradient descent (SGD) algorithms can achieve a linear iteration
speedup, they are limited significantly in practice by the communication cost,
making it difficult to achieve a linear time speedup. In this paper, we propose
a computation and communication decoupled stochastic gradient descent
(CoCoD-SGD) algorithm to run computation and communication in parallel to
reduce the communication cost. We prove that CoCoD-SGD has a linear iteration
speedup with respect to the total computation capability of the hardware
resources. In addition, it has a lower communication complexity and better time
speedup comparing with traditional distributed SGD algorithms. Experiments on
deep neural network training demonstrate the significant improvements of
CoCoD-SGD: when training ResNet18 and VGG16 with 16 Geforce GTX 1080Ti GPUs,
CoCoD-SGD is up to 2-3$\times$ faster than traditional synchronous SGD.

36
The categories of this paper are: cmp-lg
24
The categories of this paper are: dg-ga
6
The categories of this paper are: astro-ph
5
The categories of this paper are: gr-qc
19
The categories of this paper are: eess
25
The categories of this paper are: patt-sol
29
The categories of this paper are: funct-an
31
The categories of this paper are: ao-sci
30
The categories of this paper are: q-alg
28
The categories of this paper are: alg-geom
3
The categories of this paper are: physics
12
The categories of this paper are: nucl-th
11
The categories of this paper are: hep-lat
10
The categories of this paper are: q-bio
33
The categories of this paper are: plasm-ph
26
The categories of this paper are: bayes-an