Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: AutoAssist: A Framework to Accelerate Training of Deep Neural Networks   Deep neural networks have yielded superior performance in many applications;
however, the gradient computation in a deep model with millions of instances
lead to a lengthy training process even with modern GPU/TPU hardware
acceleration. In this paper, we propose AutoAssist, a simple framework to
accelerate training of a deep neural network. Typically, as the training
procedure evolves, the amount of improvement in the current model by a
stochastic gradient update on each instance varies dynamically. In AutoAssist,
we utilize this fact and design a simple instance shrinking operation, which is
used to filter out instances with relatively low marginal improvement to the
current model; thus the computationally intensive gradient computations are
performed on informative instances as much as possible. We prove that the
proposed technique outperforms vanilla SGD with existing importance sampling
approaches for linear SVM problems, and establish an O(1/k) convergence for
strongly convex problems. In order to apply the proposed techniques to
accelerate training of deep models, we propose to jointly train a very
lightweight Assistant network in addition to the original deep network referred
to as Boss. The Assistant network is designed to gauge the importance of a
given instance with respect to the current Boss such that a shrinking operation
can be applied in the batch generator. With careful design, we train the Boss
and Assistant in a nonblocking and asynchronous fashion such that overhead is
minimal. We demonstrate that AutoAssist reduces the number of epochs by 40% for
training a ResNet to reach the same test accuracy on an image classification
data set and saves 30% training time needed for a transformer model to yield
the same BLEU scores on a translation dataset.

17
The categories of this paper are: q-fin
34
The categories of this paper are: chem-ph
29
The categories of this paper are: funct-an
30
The categories of this paper are: q-alg
33
The categories of this paper are: plasm-ph
8
The categories of this paper are: hep-ex
0
The categories of this paper are: hep-ph
10
The categories of this paper are: q-bio
6
The categories of this paper are: astro-ph
32
The categories of this paper are: atom-ph
4
The categories of this paper are: cond-mat
26
The categories of this paper are: bayes-an
28
The categories of this paper are: alg-geom
37
The categories of this paper are: supr-con
31
The categories of this paper are: ao-sci
14
The categories of this paper are: math-ph