Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: The Three Stages of Learning Dynamics in High-Dimensional Kernel Methods   To understand how deep learning works, it is crucial to understand the
training dynamics of neural networks. Several interesting hypotheses about
these dynamics have been made based on empirically observed phenomena, but
there exists a limited theoretical understanding of when and why such phenomena
occur.
  In this paper, we consider the training dynamics of gradient flow on kernel
least-squares objectives, which is a limiting dynamics of SGD trained neural
networks. Using precise high-dimensional asymptotics, we characterize the
dynamics of the fitted model in two "worlds": in the Oracle World the model is
trained on the population distribution and in the Empirical World the model is
trained on a sampled dataset. We show that under mild conditions on the kernel
and $L^2$ target regression function the training dynamics undergo three stages
characterized by the behaviors of the models in the two worlds. Our theoretical
results also mathematically formalize some interesting deep learning phenomena.
Specifically, in our setting we show that SGD progressively learns more complex
functions and that there is a "deep bootstrap" phenomenon: during the second
stage, the test error of both worlds remain close despite the empirical
training error being much smaller. Finally, we give a concrete example
comparing the dynamics of two different kernels which shows that faster
training is not necessary for better generalization.

20
The categories of this paper are: acc-phys
11
The categories of this paper are: hep-lat
7
The categories of this paper are: hep-th
24
The categories of this paper are: dg-ga
21
The categories of this paper are: adap-org
10
The categories of this paper are: q-bio
30
The categories of this paper are: q-alg
18
The categories of this paper are: econ
28
The categories of this paper are: alg-geom
13
The categories of this paper are: quant-ph
35
The categories of this paper are: mtrl-th
23
The categories of this paper are: solv-int
8
The categories of this paper are: hep-ex
12
The categories of this paper are: nucl-th
36
The categories of this paper are: cmp-lg
34
The categories of this paper are: chem-ph