Deep Linear Transformers Exhibit Diverse Dynamical Behaviors

Researchers have analyzed the inference-time behavior of deep linear encoder-only transformers by modeling tokens as interacting particles. This perspective reveals that in embedding dimension two, the transformer's dynamics can be reformulated as a generalized Kuramoto-type model. This formulation, using Watanabe--Strogatz theory, shows that the dynamics are intrinsically low-dimensional and can lead to diverse long-time behaviors such as clustering, oscillations, and bifurcations, particularly for token initializations associated with the Ott--Antonsen (OA) manifold. The study also suggests these behaviors persist in higher-dimensional transformers. AI

IMPACT Provides theoretical insights into the internal dynamics of transformer models, potentially informing future architectural improvements.

RANK_REASON The cluster contains a single academic paper detailing theoretical research into transformer model dynamics. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• alphaXiv
• arXiv
• CatalyzeX Code Finder for Papers
• CORE Recommender
• DagsHub
• Gotit.pub
• Hugging Face
• IArxiv Recommender
• Ott--Antonsen (OA) manifold
• ScienceCast
• Watanabe--Strogatz theory

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →