Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Analysis of DAWNBench, a Time-to-Accuracy Machine Learning Performance
  Benchmark   Researchers have proposed hardware, software, and algorithmic optimizations
to improve the computational performance of deep learning. While some of these
optimizations perform the same operations faster (e.g., increasing GPU clock
speed), many others modify the semantics of the training procedure (e.g.,
reduced precision), and can impact the final model's accuracy on unseen data.
Due to a lack of standard evaluation criteria that considers these trade-offs,
it is difficult to directly compare these optimizations. To address this
problem, we recently introduced DAWNBench, a benchmark competition focused on
end-to-end training time to achieve near-state-of-the-art accuracy on an unseen
dataset---a combined metric called time-to-accuracy (TTA). In this work, we
analyze the entries from DAWNBench, which received optimized submissions from
multiple industrial groups, to investigate the behavior of TTA as a metric as
well as trends in the best-performing entries. We show that TTA has a low
coefficient of variation and that models optimized for TTA generalize nearly as
well as those trained using standard methods. Additionally, even though
DAWNBench entries were able to train ImageNet models in under 3 minutes, we
find they still underutilize hardware capabilities such as Tensor Cores.
Furthermore, we find that distributed entries can spend more than half of their
time on communication. We show similar findings with entries to the MLPERF v0.5
benchmark.

31
The categories of this paper are: ao-sci
13
The categories of this paper are: quant-ph
28
The categories of this paper are: alg-geom
37
The categories of this paper are: supr-con
36
The categories of this paper are: cmp-lg
26
The categories of this paper are: bayes-an
21
The categories of this paper are: adap-org
34
The categories of this paper are: chem-ph
23
The categories of this paper are: solv-int
33
The categories of this paper are: plasm-ph
25
The categories of this paper are: patt-sol
4
The categories of this paper are: cond-mat
30
The categories of this paper are: q-alg
7
The categories of this paper are: hep-th
20
The categories of this paper are: acc-phys
3
The categories of this paper are: physics