  We present a method to formulate algorithm discovery as program search, and
apply it to discover optimization algorithms for deep neural network training.
We leverage efficient search techniques to explore an infinite and sparse
program space. To bridge the large generalization gap between proxy and target
tasks, we also introduce program selection and simplification strategies. Our
method discovers a simple and effective optimization algorithm, $\textbf{Lion}$
($\textit{Evo$\textbf{L}$ved S$\textbf{i}$gn M$\textbf{o}$me$\textbf{n}$tum}$).
It is more memory-efficient than Adam as it only keeps track of the momentum.
Different from adaptive optimizers, its update has the same magnitude for each
parameter calculated through the sign operation. We compare Lion with widely
used optimizers, such as Adam and Adafactor, for training a variety of models
on different tasks. On image classification, Lion boosts the accuracy of ViT by
up to 2% on ImageNet and saves up to 5x the pre-training compute on JFT. On
vision-language contrastive learning, we achieve 88.3% $\textit{zero-shot}$ and
91.1% $\textit{fine-tuning}$ accuracy on ImageNet, surpassing the previous best
results by 2% and 0.1%, respectively. On diffusion models, Lion outperforms
Adam by achieving a better FID score and reducing the training compute by up to
2.3x. For autoregressive, masked language modeling, and fine-tuning, Lion
exhibits a similar or better performance compared to Adam. Our analysis of Lion
reveals that its performance gain grows with the training batch size. It also
requires a smaller learning rate than Adam due to the larger norm of the update
produced by the sign function. Additionally, we examine the limitations of Lion
and identify scenarios where its improvements are small or not statistically
significant. Lion is also successfully deployed in production systems such as
Google search ads CTR model.
