Researchers have developed a new optimization technique called Periodic Row-wise Muon to improve the training efficiency of Diffusion Transformers (DiTs). This method builds upon the original Muon optimizer by performing full spectral updates less frequently and using a more computationally efficient row-wise update for the remaining steps. A co-designed distributed implementation further enhances speed by operating directly on sharded momentum and overlapping computation with communication. The new technique maintains or slightly improves generative quality while significantly reducing optimizer time, step time, and communication volume, making it more efficient for training large DiTs. AI
IMPACT Enhances training efficiency for large diffusion models, potentially accelerating research and development in generative AI.
RANK_REASON Academic paper detailing a novel optimization technique for AI models. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
