Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Exploring Hidden Dimensions in Parallelizing Convolutional Neural
  Networks   The past few years have witnessed growth in the computational requirements
for training deep convolutional neural networks. Current approaches parallelize
training onto multiple devices by applying a single parallelization strategy
(e.g., data or model parallelism) to all layers in a network. Although easy to
reason about, these approaches result in suboptimal runtime performance in
large-scale distributed training, since different layers in a network may
prefer different parallelization strategies. In this paper, we propose
layer-wise parallelism that allows each layer in a network to use an individual
parallelization strategy. We jointly optimize how each layer is parallelized by
solving a graph search problem. Our evaluation shows that layer-wise
parallelism outperforms state-of-the-art approaches by increasing training
throughput, reducing communication costs, achieving better scalability to
multiple GPUs, while maintaining original network accuracy.

27
The categories of this paper are: comp-gas
35
The categories of this paper are: mtrl-th
0
The categories of this paper are: hep-ph
2
The categories of this paper are: math
22
The categories of this paper are: chao-dyn
6
The categories of this paper are: astro-ph
11
The categories of this paper are: hep-lat
26
The categories of this paper are: bayes-an
20
The categories of this paper are: acc-phys
23
The categories of this paper are: solv-int
29
The categories of this paper are: funct-an
9
The categories of this paper are: nlin
37
The categories of this paper are: supr-con
28
The categories of this paper are: alg-geom
15
The categories of this paper are: nucl-ex
3
The categories of this paper are: physics