Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: TensorOpt: Exploring the Tradeoffs in Distributed DNN Training with
  Auto-Parallelism   A good parallelization strategy can significantly improve the efficiency or
reduce the cost for the distributed training of deep neural networks (DNNs).
Recently, several methods have been proposed to find efficient parallelization
strategies but they all optimize a single objective (e.g., execution time,
memory consumption) and produce only one strategy. We propose FT, an efficient
algorithm that searches for an optimal set of parallelization strategies to
allow the trade-off among different objectives. FT can adapt to different
scenarios by minimizing the memory consumption when the number of devices is
limited and fully utilize additional resources to reduce the execution time.
For popular DNN models (e.g., vision, language), an in-depth analysis is
conducted to understand the trade-offs among different objectives and their
influence on the parallelization strategies. We also develop a user-friendly
system, called TensorOpt, which allows users to run their distributed DNN
training jobs without caring the details of parallelization strategies.
Experimental results show that FT runs efficiently and provides accurate
estimation of runtime costs, and TensorOpt is more flexible in adapting to
resource availability compared with existing frameworks.

30
The categories of this paper are: q-alg
21
The categories of this paper are: adap-org
11
The categories of this paper are: hep-lat
35
The categories of this paper are: mtrl-th
25
The categories of this paper are: patt-sol
5
The categories of this paper are: gr-qc
19
The categories of this paper are: eess
3
The categories of this paper are: physics
17
The categories of this paper are: q-fin
8
The categories of this paper are: hep-ex
2
The categories of this paper are: math
0
The categories of this paper are: hep-ph
4
The categories of this paper are: cond-mat
29
The categories of this paper are: funct-an
32
The categories of this paper are: atom-ph
7
The categories of this paper are: hep-th