Interventional data fails to teach language models causal direction in Simpson's paradox scenarios

A new research paper explores the effectiveness of interventional data in teaching language models causal reasoning. The study found that in scenarios exhibiting Simpson's paradox, where observational correlation and causal effect have opposite signs, increasing interventional samples during pretraining did not improve the model's ability to discern causal direction. Instead, the model's inference-time context heavily influenced its interpretation, with purely observational contexts leading to systematic sign reversals. The research suggests that while the capability for causal inference resides in the model's weights, its activation is controlled by the inference-time context, particularly in the middle layers. AI

IMPACT Challenges the assumption that interventional data is superior for teaching causal reasoning to LLMs, suggesting context plays a critical role.

RANK_REASON Research paper detailing findings on language model causal reasoning capabilities. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• alphaXiv
• CatalyzeX
• Cladder & Jansen
• DagsHub
• Gotit.pub
• Hugging Face
• Language Models
• ScienceCast
• Simpson's paradox

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →