Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Gradient Sparification for Asynchronous Distributed Training   Modern large scale machine learning applications require stochastic
optimization algorithms to be implemented on distributed computational
architectures. A key bottleneck is the communication overhead for exchanging
information, such as stochastic gradients, among different nodes. Recently,
gradient sparsification techniques have been proposed to reduce communications
cost and thus alleviate the network overhead. However, most of gradient
sparsification techniques consider only synchronous parallelism and cannot be
applied in asynchronous scenarios, such as asynchronous distributed training
for federated learning at mobile devices.
  In this paper, we present a dual-way gradient sparsification approach (DGS)
that is suitable for asynchronous distributed training. We let workers download
model difference, instead of the global model, from the server, and the model
difference information is also sparsified so that the information exchanged
overhead is reduced by sparsifying the dual-way communication between the
server and workers. To preserve accuracy under dual-way sparsification, we
design a sparsification aware momentum (SAMomentum) to turn sparsification into
adaptive batch size between each parameter. We conduct experiments at a cluster
of 32 workers, and the results show that, with the same compression ratio but
much lower communication cost, our approach can achieve better scalability and
generalization ability.

15
The categories of this paper are: nucl-ex
21
The categories of this paper are: adap-org
28
The categories of this paper are: alg-geom
22
The categories of this paper are: chao-dyn
0
The categories of this paper are: hep-ph
32
The categories of this paper are: atom-ph
34
The categories of this paper are: chem-ph
2
The categories of this paper are: math
9
The categories of this paper are: nlin
24
The categories of this paper are: dg-ga
18
The categories of this paper are: econ
35
The categories of this paper are: mtrl-th
11
The categories of this paper are: hep-lat
17
The categories of this paper are: q-fin
20
The categories of this paper are: acc-phys
26
The categories of this paper are: bayes-an