New Theory Explains Orthogonalization's Effectiveness in Neural Networks

Researchers have developed a new theoretical framework to explain why orthogonalization techniques, such as Muon, are effective in optimizing neural networks. The study reveals that the Hessian of a neural network, which represents the optimal preconditioner, exhibits a row-block diagonally dominant structure. This structure is driven by the overparameterization of weight matrices, a phenomenon observed in models like language models. The analysis, conducted at initialization, provides quantitative asymptotic theory for various canonical models, including deep linear networks and ReLU networks, confirming that diagonal entries of the gradient self outer-product grow faster than off-diagonal ones. AI

IMPACT Provides theoretical grounding for optimization techniques, potentially leading to more efficient training of large language models.

RANK_REASON The cluster contains a research paper detailing theoretical analysis of neural network optimization. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• AdamW
• arXiv
• Deep Linear Networks
• Hessian
• Language Models
• linear attention
• Muon
• Relu Networks

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →