Arthur Castello Branco De Oliveira

AF-Muon optimizer eliminates AdamW for tied-embedding models, saving memory

Researchers have developed AF-Muon, an optimizer that removes the need for AdamW in tied-embedding models, potentially saving around 20% of optimizer-state memory. This new approach maintains Muon's spectral-norm steepe…