Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: dPRO: A Generic Profiling and Optimization System for Expediting
  Distributed DNN Training   Distributed training using multiple devices (i.e., GPU servers) has been
widely adopted for learning DNN models over large datasets. However, the
performance of large-scale distributed training tends to be far from linear
speed-up in practice. Given the complexity of distributed systems, it is
challenging to identify the root cause(s) of inefficiency and exercise
effective performance optimizations when unexpected low training speed occurs.
To date, there exists no software tool which diagnoses performance issues and
helps expedite distributed DNN training, while the training can be run using
different machine learning frameworks. This paper proposes dPRO, a toolkit that
includes: (1) an efficient profiler that collects runtime traces of distributed
DNN training across multiple frameworks, especially fine-grained communication
traces, and constructs global data flow graphs including detailed communication
operations for accurate replay; (2) an optimizer that effectively identifies
performance bottlenecks and explores optimization strategies (from computation,
communication and memory aspects) for training acceleration. We implement dPRO
on multiple deep learning frameworks (PyTorch, TensorFlow, MXNet) and
representative communication schemes (AllReduce and Parameter Server
architecture). Extensive experiments show that dPRO predicts performance of
distributed training in various settings with<5% errors in most cases and finds
optimization strategies with up to87.1%speed-up over the baselines.

32
The categories of this paper are: atom-ph
8
The categories of this paper are: hep-ex
9
The categories of this paper are: nlin
35
The categories of this paper are: mtrl-th
33
The categories of this paper are: plasm-ph
17
The categories of this paper are: q-fin
25
The categories of this paper are: patt-sol
4
The categories of this paper are: cond-mat
16
The categories of this paper are: stat
5
The categories of this paper are: gr-qc
12
The categories of this paper are: nucl-th
19
The categories of this paper are: eess
37
The categories of this paper are: supr-con
10
The categories of this paper are: q-bio
15
The categories of this paper are: nucl-ex
23
The categories of this paper are: solv-int