Modular Addition

Muon-trained transformers exhibit post-grokking collapse, losing generalization

A new research paper explores a phenomenon termed "post-grokking collapse" in transformers trained with the Muon optimizer. While Muon initially demonstrates faster learning on modular addition tasks, the models eventua…