Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Backward Feature Correction: How Deep Learning Performs Deep Learning   How does a 110-layer ResNet learn a high-complexity classifier using
relatively few training examples and short training time? We present a theory
towards explaining this in terms of Hierarchical Learning. We refer
hierarchical learning as the learner learns to represent a complicated target
function by decomposing it into a sequence of simpler functions to reduce
sample and time complexity. We formally analyze how multi-layer neural networks
can perform such hierarchical learning efficiently and automatically by
applying SGD.
  On the conceptual side, we present, to the best of our knowledge, the FIRST
theory result indicating how deep neural networks can still be sample and time
efficient using SGD on certain hierarchical learning tasks, when NO KNOWN
existing algorithm is efficient. We establish a new principle called "backward
feature correction", where training higher-level layers in the network can
improve the features of lower-level ones. We believe this is the key to
understand the deep learning process in multi-layer neural networks.
  On the technical side, we show for regression and even binary classification,
for every input dimension $d>0$, there is a concept class of degree $\omega(1)$
polynomials so that, using $\omega(1)$-layer neural networks as learners, SGD
can learn any function from this class in $\mathsf{poly}(d)$ time and sample
complexity to any $\frac{1}{\mathsf{poly}(d)}$ error, through learning to
represent it as a composition of $\omega(1)$ layers of quadratic functions. In
contrast, we do not know any other simple algorithm (including layer-wise
training or applying kernel method sequentially) that can learn this concept
class in $\mathsf{poly}(d)$ time even to any $d^{-0.01}$ error. As a side
result, we prove $d^{\omega(1)}$ lower bounds for several non-hierarchical
learners, including any kernel methods, neural tangent or neural compositional
kernels.

33
The categories of this paper are: plasm-ph
30
The categories of this paper are: q-alg
13
The categories of this paper are: quant-ph
9
The categories of this paper are: nlin
22
The categories of this paper are: chao-dyn
25
The categories of this paper are: patt-sol
35
The categories of this paper are: mtrl-th
11
The categories of this paper are: hep-lat
23
The categories of this paper are: solv-int
5
The categories of this paper are: gr-qc
31
The categories of this paper are: ao-sci
7
The categories of this paper are: hep-th
12
The categories of this paper are: nucl-th
10
The categories of this paper are: q-bio
21
The categories of this paper are: adap-org
37
The categories of this paper are: supr-con