A new arXiv paper explores the concept of "internalization" in neural networks, where systems learn to perform computational procedures by embedding them into their own weights. Researchers investigated how transformers internalize chain-of-thought (CoT) tokens and found that while this process aids learning, it can degrade out-of-distribution performance. The paper provides a theoretical analysis of successful internalization for parity learning tasks, demonstrating how a simplified transformer can learn to directly compute parities after explicit CoT supervision, relating this to the phenomenon of Positive Distribution Shift. AI
IMPACT This research could lead to more efficient and robust neural network training methods by understanding how models internalize procedures.
RANK_REASON The cluster contains a research paper published on arXiv detailing theoretical analysis of neural network learning processes. [lever_c_demoted from research: ic=1 ai=1.0]
- alphaXiv
- arXiv
- CatalyzeX Code Finder for Papers
- CORE Recommender
- DagsHub
- Gotit.pub
- Hugging Face
- IArxiv Recommender
- Med+26
- Nikolaos Tsilivis
- ScienceCast
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →