  We present a smoothly broken power law functional form (that we refer to as a
Broken Neural Scaling Law (BNSL)) that accurately models & extrapolates the
scaling behaviors of deep neural networks (i.e. how the evaluation metric of
interest varies as amount of compute used for training (or inference), number
of model parameters, training dataset size, model input size, number of
training steps, or upstream performance varies) for various architectures & for
each of various tasks within a large & diverse set of upstream & downstream
tasks, in zero-shot, prompted, & finetuned settings. This set includes
large-scale vision, language, audio, video, diffusion, generative modeling,
multimodal learning, contrastive learning, AI alignment, AI capabilities,
robotics, out-of-distribution (OOD) generalization, continual learning,
transfer learning, uncertainty estimation / calibration, OOD detection,
adversarial robustness, distillation, sparsity, retrieval, quantization,
pruning, fairness, molecules, computer programming/coding, math word problems,
"emergent phase transitions", arithmetic, supervised learning,
unsupervised/self-supervised learning, & reinforcement learning (single agent &
multi-agent). When compared to other functional forms for neural scaling, this
functional form yields extrapolations of scaling behavior that are considerably
more accurate on this set. Moreover, this functional form accurately models &
extrapolates scaling behavior that other functional forms are incapable of
expressing such as the nonmonotonic transitions present in the scaling behavior
of phenomena such as double descent & the delayed, sharp inflection points
present in the scaling behavior of tasks such as arithmetic. Lastly, we use
this functional form to glean insights about the limit of the predictability of
scaling behavior. Code is available at
https://github.com/ethancaballero/broken_neural_scaling_laws
