memory overheads associated with adaptive gradient methods like Adam and LAMB, which are commonly used in training LLMs.

The authors first identify a problem with existing memory-efficient optimizers like Adafactor, which drastically reduce memory usage but suffer from performance degradation. They attribute this to the instability introduced by the non-negative matrix factorization operation in Adafactor, which they term as "erroneous update".

To address this, they propose a confidence-guided strategy th