Researchers have identified a significant shortcut in how small language models perform arithmetic tasks using chain-of-thought prompting. Instead of relying on logical step-by-step reasoning, these models primarily copy the numerical value that appears last in the sequence before the answer delimiter. This positional copying mechanism accounts for the vast majority of their accuracy, even when intermediate steps are incorrect or shuffled. The study suggests that current evaluation methods for CoT faithfulness may be conflating this positional answer transport with genuine computational ability, potentially misrepresenting model understanding. AI
Summary written by gemini-2.5-flash-lite from 1 sources. How we write summaries →
IMPACT Reveals a critical flaw in how small language models handle arithmetic, potentially invalidating current CoT evaluation methods and impacting the reliability of AI in quantitative tasks.
RANK_REASON The cluster contains an academic paper detailing a novel finding about language model behavior. [lever_c_demoted from research: ic=1 ai=1.0]