Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Optimal Complexity in Decentralized Training   Decentralization is a promising method of scaling up parallel machine
learning systems. In this paper, we provide a tight lower bound on the
iteration complexity for such methods in a stochastic non-convex setting. Our
lower bound reveals a theoretical gap in known convergence rates of many
existing decentralized training algorithms, such as D-PSGD. We prove by
construction this lower bound is tight and achievable. Motivated by our
insights, we further propose DeTAG, a practical gossip-style decentralized
algorithm that achieves the lower bound with only a logarithm gap. Empirically,
we compare DeTAG with other decentralized algorithms on image classification
tasks, and we show DeTAG enjoys faster convergence compared to baselines,
especially on unshuffled data and in sparse networks.

34
The categories of this paper are: chem-ph
18
The categories of this paper are: econ
0
The categories of this paper are: hep-ph
14
The categories of this paper are: math-ph
7
The categories of this paper are: hep-th
37
The categories of this paper are: supr-con
27
The categories of this paper are: comp-gas
35
The categories of this paper are: mtrl-th
30
The categories of this paper are: q-alg
15
The categories of this paper are: nucl-ex
32
The categories of this paper are: atom-ph
3
The categories of this paper are: physics
28
The categories of this paper are: alg-geom
29
The categories of this paper are: funct-an
24
The categories of this paper are: dg-ga
10
The categories of this paper are: q-bio