  We propose that the grokking phenomenon, where the train loss of a neural
network decreases much earlier than its test loss, can arise due to a neural
network transitioning from lazy training dynamics to a rich, feature learning
regime. To illustrate this mechanism, we study the simple setting of vanilla
gradient descent on a polynomial regression problem with a two layer neural
network which exhibits grokking without regularization in a way that cannot be
explained by existing theories. We identify sufficient statistics for the test
loss of such a network, and tracking these over training reveals that grokking
arises in this setting when the network first attempts to fit a kernel
regression solution with its initial features, followed by late-time feature
learning where a generalizing solution is identified after train loss is
already low. We provide an asymptotic theoretical description of the grokking
dynamics in this model using dynamical mean field theory (DMFT) for high
dimensional data. We find that the key determinants of grokking are the rate of
feature learning -- which can be controlled precisely by parameters that scale
the network output -- and the alignment of the initial features with the target
function $y(x)$. We argue this delayed generalization arises when (1) the top
eigenvectors of the initial neural tangent kernel and the task labels $y(x)$
are misaligned, but (2) the dataset size is large enough so that it is possible
for the network to generalize eventually, but not so large that train loss
perfectly tracks test loss at all epochs, and (3) the network begins training
in the lazy regime so does not learn features immediately. We conclude with
evidence that this transition from lazy (linear model) to rich training
(feature learning) can control grokking in more general settings, like on
MNIST, one-layer Transformers, and student-teacher networks.
