  Robust generalization is a major challenge in deep learning, particularly
when the number of trainable parameters is very large. In general, it is very
difficult to know if the network has memorized a particular set of examples or
understood the underlying rule (or both). Motivated by this challenge, we study
an interpretable model where generalizing representations are understood
analytically, and are easily distinguishable from the memorizing ones. Namely,
we consider two-layer neural networks trained on modular arithmetic tasks where
($\xi \cdot 100\%$) of labels are corrupted (\emph{i.e.} some results of the
modular operations in the training set are incorrect). We show that (i) it is
possible for the network to memorize the corrupted labels \emph{and} achieve
$100\%$ generalization at the same time; (ii) the memorizing neurons can be
identified and pruned, lowering the accuracy on corrupted data and improving
the accuracy on uncorrupted data; (iii) regularization methods such as weight
decay, dropout and BatchNorm force the network to ignore the corrupted data
during optimization, and achieve $100\%$ accuracy on the uncorrupted dataset;
and (iv) the effect of these regularization methods is (``mechanistically'')
interpretable: weight decay and dropout force all the neurons to learn
generalizing representations, while BatchNorm de-amplifies the output of
memorizing neurons and amplifies the output of the generalizing ones. Finally,
we show that in the presence of regularization, the training dynamics involves
two consecutive stages: first, the network undergoes the \emph{grokking}
dynamics reaching high train \emph{and} test accuracy; second, it unlearns the
memorizing representations, where train accuracy suddenly jumps from $100\%$ to
$100 (1-\xi)\%$.
