CausalT5k: Diagnosing Refusal and Failure Modes in Trustworthy Causal Reasoning Across Causal Rungs
Researchers have introduced CausalT5k (CTK), a new diagnostic benchmark designed to identify specific failure modes in large language models' causal reasoning capabilities. CTK comprises over 5,000 cases across 10 domains and addresses all three levels of Pearl's Ladder of Causation. Unlike existing benchmarks that focus solely on correctness, CTK annotates causal rungs, trap types, pressure sensitivity, and refusal quality to reveal why a model fails. The benchmark aims to provide a substrate for studying these nuanced causal reasoning failure profiles. AI
IMPACT Provides a new diagnostic tool to better understand and address specific failure modes in LLM causal reasoning.