New theory shows Gaussian noise preserves spectral law in multi-head self-attention

Researchers have established a theoretical framework called Gaussian Equivalence for multi-head self-attention mechanisms. This framework demonstrates that replacing standard softmax attention with rescaled scores plus Gaussian noise maintains the limiting spectral law of the centered output. The findings, derived using random matrix theory, also extend to value and output projections dependent on keys, differentiating the impact of head allocation and projection widths on spectrum preservation. AI

IMPACT Provides a deeper theoretical understanding of a core component in modern neural networks, potentially guiding future architectural designs.

RANK_REASON Academic paper detailing a theoretical advance in neural network mechanisms. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• Gaussian Equivalence
• Random Matrix Theory

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →