Study reveals critical role of attention residual in transformer models

A new research paper investigates the impact of residual connections in Pre-LN GPT-style transformers. The study found that removing the attention residual connection consistently leads to performance collapse, while removing the feed-forward residual connection shows a reproducible recovery effect at a smaller scale, though its behavior at a larger scale remains uncertain due to variance. The researchers also identified and corrected an experimental measurement confound, proposing a cross-position routing hypothesis to explain their findings and releasing all code and data to ensure reproducibility. AI

IMPACT This research clarifies architectural choices in transformer models, potentially guiding future model development and optimization.

RANK_REASON The cluster contains a research paper detailing experimental findings on transformer architectures. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• alphaXiv
• arXiv
• AttnOnly
• CatalyzeX
• DagsHub
• FFNOnly
• generative pre-trained transformer
• Gotit.pub
• Hugging Face
• Pre-LN Transformers
• ScienceCast

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →