New SHAM-Bench benchmark evaluates LLM hallucination detection across tasks

Researchers have introduced SHAM-Bench, a new benchmark designed to evaluate span-level hallucination detection in large language models across multiple natural language generation tasks. The benchmark aggregates 6,461 examples from existing datasets under a common schema and evaluation protocol. Initial evaluations using SHAM-Bench revealed that while in-domain fine-tuned detectors perform well, they degrade under distribution shifts, whereas prompt-based methods with reasoning demonstrate greater robustness across different tasks. AI

IMPACT This benchmark aims to improve the detection and mitigation of factual inaccuracies in LLM outputs, leading to more reliable AI systems.

RANK_REASON The cluster describes a new academic paper introducing a benchmark dataset for evaluating AI models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• Hugging Face
• natural language generation
• SHAM-Bench

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →