Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Layer-Parallel Training of Residual Networks with Auxiliary-Variable
  Networks   Gradient-based methods for the distributed training of residual networks
(ResNets) typically require a forward pass of the input data, followed by
back-propagating the error gradient to update model parameters, which becomes
time-consuming as the network goes deeper. To break the algorithmic locking and
exploit synchronous module parallelism in both the forward and backward modes,
auxiliary-variable methods have attracted much interest lately but suffer from
significant communication overhead and lack of data augmentation. In this work,
a novel joint learning framework for training realistic ResNets across multiple
compute devices is established by trading off the storage and recomputation of
external auxiliary variables. More specifically, the input data of each
independent processor is generated from its low-capacity auxiliary network
(AuxNet), which permits the use of data augmentation and realizes forward
unlocking. The backward passes are then executed in parallel, each with a local
loss function that originates from the penalty or augmented Lagrangian (AL)
methods. Finally, the proposed AuxNet is employed to reproduce the updated
auxiliary variables through an end-to-end training process. We demonstrate the
effectiveness of our methods on ResNets and WideResNets across CIFAR-10,
CIFAR-100, and ImageNet datasets, achieving speedup over the traditional
layer-serial training method while maintaining comparable testing accuracy.

30
The categories of this paper are: q-alg
27
The categories of this paper are: comp-gas
17
The categories of this paper are: q-fin
36
The categories of this paper are: cmp-lg
15
The categories of this paper are: nucl-ex
25
The categories of this paper are: patt-sol
5
The categories of this paper are: gr-qc
22
The categories of this paper are: chao-dyn
21
The categories of this paper are: adap-org
34
The categories of this paper are: chem-ph
13
The categories of this paper are: quant-ph
7
The categories of this paper are: hep-th
4
The categories of this paper are: cond-mat
23
The categories of this paper are: solv-int
10
The categories of this paper are: q-bio
26
The categories of this paper are: bayes-an