Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: On Communication Compression for Distributed Optimization on
  Heterogeneous Data   Lossy gradient compression, with either unbiased or biased compressors, has
become a key tool to avoid the communication bottleneck in centrally
coordinated distributed training of machine learning models. We analyze the
performance of two standard and general types of methods: (i) distributed
quantized SGD (D-QSGD) with arbitrary unbiased quantizers and (ii) distributed
SGD with error-feedback and biased compressors (D-EF-SGD) in the heterogeneous
(non-iid) data setting. Our results indicate that D-EF-SGD is much less
affected than D-QSGD by non-iid data, but both methods can suffer a slowdown if
data-skewness is high. We further study two alternatives that are not (or much
less) affected by heterogenous data distributions: first, a recently proposed
method that is effective on strongly convex problems, and secondly, we point
out a more general approach that is applicable to linear compressors only but
effective in all considered scenarios.

31
The categories of this paper are: ao-sci
35
The categories of this paper are: mtrl-th
0
The categories of this paper are: hep-ph
29
The categories of this paper are: funct-an
4
The categories of this paper are: cond-mat
8
The categories of this paper are: hep-ex
9
The categories of this paper are: nlin
18
The categories of this paper are: econ
37
The categories of this paper are: supr-con
10
The categories of this paper are: q-bio
25
The categories of this paper are: patt-sol
11
The categories of this paper are: hep-lat
12
The categories of this paper are: nucl-th
32
The categories of this paper are: atom-ph
3
The categories of this paper are: physics
28
The categories of this paper are: alg-geom