A new research paper explores the mechanisms behind deductive reasoning in language models, distinguishing between horizontal and vertical reasoning. The study, which trained small transformer models on symbolic implication tasks, found that Chain-of-Thought supervision helps models learn rule-based inference rather than relying on statistical shortcuts. For horizontal reasoning, models developed interpretable circuits for rule completion and decision-making, while for vertical reasoning, Chain-of-Thought acted more as a curriculum learning tool to acquire complex patterns. AI
IMPACT Provides a low-level account of how transformers implement deductive reasoning, suggesting different functions for Chain-of-Thought in horizontal versus vertical reasoning.
RANK_REASON Research paper published on arXiv detailing findings about language model reasoning. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
