Researchers analyze signal propagation in normalization-free transformers

Researchers have analyzed signal propagation in normalization-free transformers using the averaged partial Jacobian norm (APJN). Their theory explains how attention mechanisms affect APJN growth in deep vision transformers. The study indicates that transformers with LayerNorm exhibit power-law APJN growth, while those using elementwise nonlinearities are subcritical, requiring careful initialization and optimization for stable training. AI

IMPACT Provides theoretical insights into transformer training stability, potentially guiding future architecture design.

RANK_REASON Academic paper analyzing signal propagation in transformer architectures. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv stat.ML →

• arXiv
• Derf
• Dynamic erf
• Dynamic Tanh
• Sergey Alekseev

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →