Researchers have identified that the computational cost of looped transformers during inference is determined by their dynamical phase, which is influenced by initialization. Networks with identical architectures and objectives can enter distinct dynamical phases, leading to different compute scaling behaviors. The study details how specific bifurcation mechanisms, such as saddle-node folds and Neimark-Sacker transitions, differentiate these phases and impact test-time compute. AI
IMPACT This research could lead to more efficient transformer architectures by understanding how initialization impacts compute scaling.
RANK_REASON This is a research paper published on arXiv detailing a new finding about the behavior of looped transformers. [lever_c_demoted from research: ic=1 ai=1.0]
- alphaXiv
- arXiv
- CatalyzeX Code Finder for Papers
- condensed matter physics
- CORE Recommender
- cs.LG
- DagsHub
- Disordered Systems and Neural Networks
- Gotit.pub
- Hugging Face
- IArxiv Recommender
- Influence Flower
- Looped Transformers
- ScienceCast
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
