Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Automatic Cross-Replica Sharding of Weight Update in Data-Parallel
  Training   In data-parallel synchronous training of deep neural networks, different
devices (replicas) run the same program with different partitions of the
training batch, but weight update computation is repeated on all replicas,
because the weights do not have a batch dimension to partition. This can be a
bottleneck for performance and scalability in typical language models with
large weights, and models with small per-replica batch size which is typical in
large-scale training. This paper presents an approach to automatically shard
the weight update computation across replicas with efficient communication
primitives and data formatting, using static analysis and transformations on
the training computation graph. We show this technique achieves substantial
speedups on typical image and language models on Cloud TPUs, requiring no
change to model code. This technique helps close the gap between traditionally
expensive (ADAM) and cheap (SGD) optimizers, as they will only take a small
part of training step time and have similar peak memory usage. It helped us to
achieve state-of-the-art training performance in Google's MLPerf 0.6
submission.

18
The categories of this paper are: econ
20
The categories of this paper are: acc-phys
13
The categories of this paper are: quant-ph
28
The categories of this paper are: alg-geom
21
The categories of this paper are: adap-org
25
The categories of this paper are: patt-sol
7
The categories of this paper are: hep-th
35
The categories of this paper are: mtrl-th
27
The categories of this paper are: comp-gas
24
The categories of this paper are: dg-ga
22
The categories of this paper are: chao-dyn
3
The categories of this paper are: physics
12
The categories of this paper are: nucl-th
23
The categories of this paper are: solv-int
19
The categories of this paper are: eess
10
The categories of this paper are: q-bio