  The remarkable practical success of deep learning has revealed some major
surprises from a theoretical perspective. In particular, simple gradient
methods easily find near-optimal solutions to non-convex optimization problems,
and despite giving a near-perfect fit to training data without any explicit
effort to control model complexity, these methods exhibit excellent predictive
accuracy. We conjecture that specific principles underlie these phenomena: that
overparametrization allows gradient methods to find interpolating solutions,
that these methods implicitly impose regularization, and that
overparametrization leads to benign overfitting. We survey recent theoretical
progress that provides examples illustrating these principles in simpler
settings. We first review classical uniform convergence results and why they
fall short of explaining aspects of the behavior of deep learning methods. We
give examples of implicit regularization in simple settings, where gradient
methods lead to minimal norm functions that perfectly fit the training data.
Then we review prediction methods that exhibit benign overfitting, focusing on
regression problems with quadratic loss. For these methods, we can decompose
the prediction rule into a simple component that is useful for prediction and a
spiky component that is useful for overfitting but, in a favorable setting,
does not harm prediction accuracy. We focus specifically on the linear regime
for neural networks, where the network can be approximated by a linear model.
In this regime, we demonstrate the success of gradient flow, and we consider
benign overfitting with two-layer networks, giving an exact asymptotic analysis
that precisely demonstrates the impact of overparametrization. We conclude by
highlighting the key challenges that arise in extending these insights to
realistic deep learning settings.
