Research: CoT unfaithfulness detection fails on incorrect model answers

A new research paper published on arXiv explores the unfaithfulness of Chain-of-Thought (CoT) reasoning in large language models. The study, titled "Two Regimes of Chain-of-Thought Unfaithfulness: Behavioral Detection Fails Where Models Are Wrong," found that answer correctness significantly impacts the ability to detect unfaithful reasoning. Behavioral detection methods, which analyze model outputs, perform poorly when the model's final answer is incorrect, a scenario where most unfaithfulness occurs. The research also highlighted that standard metrics like step-removal do not accurately correlate with human annotations of unfaithfulness and that different models exhibit distinct behaviors in their reasoning processes. AI

IMPACT Highlights limitations in current methods for verifying LLM reasoning, suggesting a need for more robust auditing techniques.

RANK_REASON Research paper published on arXiv detailing findings on LLM reasoning faithfulness. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• alphaXiv
• arXiv
• CORE Recommender
• DagsHub
• FaithCoT-Bench
• Gotit.pub
• Hugging Face
• Llama-3.1:8b
• Qwen 2.5 7B
• ScienceCast

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →