Transformer theory extended to include feed-forward networks

Researchers have developed an extended dynamical theory for Transformers that incorporates the feed-forward network (FFN) as a local steering field. This new theory suggests that the tangential component of the FFN is crucial for movement in residual-direction space and that layers with small commutator defects can be parallelized. Experiments across GPT-2, Pythia, Mistral, and Llama models show that this extended theory improves one-step angular prediction, with the FFN's contribution increasing in later models like Llama 3-8B. Intervention experiments confirm that the tangential FFN component is vital for maintaining model quality and output diversity. AI

IMPACT Provides a deeper theoretical understanding of Transformer architecture, potentially guiding future model development and optimization.

RANK_REASON The cluster contains a research paper detailing a new theoretical framework for understanding Transformer models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• feed-forward network
• GPT-2
• Llama
• Llama 3-8B
• Pythia
• Transformer

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →