algorithm calculates an instability matrix at each training step and applies non-negative matrix factorization to it, storing only the moving averages of these factors rather than the full matrix, thus saving considerable memory footprint.

The authors validate their algorithm through extensive experiments on training tasks of three large language models: BERT, GPT-2, and T5. The results show that CAME achieves faster convergence and better generalization capability than Adam in BERT pre-training task with 