  This paper introduces SparseOptimizer, a novel deep learning optimizer that
exploits Moreau-Yosida regularization to naturally induce sparsity in large
language models such as BERT, ALBERT and GPT. Key to the design of
SparseOptimizer is an embedded shrinkage operator, which imparts sparsity
directly within the optimization process. This operator, backed by a sound
theoretical framework, includes an analytical solution, thereby reinforcing the
optimizer's robustness and efficacy. Crucially, SparseOptimizer's plug-and-play
functionality eradicates the need for code modifications, making it a
universally adaptable tool for a wide array of large language models. Empirical
evaluations on benchmark datasets such as GLUE, RACE, SQuAD1, and SQuAD2
confirm that SparseBERT and SparseALBERT, when sparsified using
SparseOptimizer, achieve performance comparable to their dense counterparts,
BERT and ALBERT, while significantly reducing their parameter count. Further,
this work proposes an innovative optimizer-compiler co-design strategy,
demonstrating the potential of inference acceleration (\textbf{3.37x},
\textbf{6.30x}, and \textbf{7.15x} in comparison with Pytorch, TensorFlow, and
LLVM generic compile, respectively) in SparseBERT when paired with an
appropriately designed compiler. This study represents a significant step
forward in the evolution of efficient, scalable, and high-performing large
language models, setting a precedent for future exploration and optimization in
this domain. The SparseOptimizer code and SparseALBERT model will be publicly
available upon paper acceptance.
