A new research paper titled "Shattered Compositionality" explores the learning dynamics of transformers when trained on arithmetic tasks. The study, led by Xingyu Zhao, found that these models often acquire skills in reverse or parallel orders, deviating from human-like sequential learning. This "shattered compositionality" is attributed to correlational matching with training data rather than causal reasoning, leading to characteristic mixing errors and reduced robustness. The research indicates that these issues persist even in modern large language models and are not resolved by scaling or scratchpad supervision. AI
IMPACT Highlights limitations in transformer arithmetic reasoning and robustness, suggesting current training methods may not foster human-like compositional skills.
RANK_REASON Research paper detailing novel findings on transformer learning dynamics. [lever_c_demoted from research: ic=1 ai=1.0]
- alphaXiv
- arXiv
- CatalyzeX
- DagsHub
- Gotit.pub
- Hugging Face
- large-language models
- ScienceCast
- transformers
- Xingyu Zhao
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
