Dense local dependencies cause Transformer training instability in long sequences

A new research paper identifies dense local dependencies as a primary cause of training instability in autoregressive transformer language models when processing long sequences, especially with low-precision arithmetic. The study, published on arXiv, explains that these dependencies create a high-rank attention structure that requires increasingly large logits for approximation as sequence length grows, leading to instability. The findings suggest that explicitly modeling dense local dependencies is crucial for developing more stable and scalable long-context transformer architectures. AI

IMPACT Suggests a design principle for more stable and scalable long-context transformer architectures.

RANK_REASON Research paper detailing a technical finding about Transformer training instability. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• alphaXiv
• arXiv
• CatalyzeX
• DagsHub
• Gotit.pub
• Hugging Face
• IArxiv
• ScienceCast
• Suvadeep Hajra
• Transformer

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →