New ACT-Eval framework reveals pervasive hallucinations in LLM chess commentary

A new evaluation framework called ACT-Eval has been developed to assess the factual accuracy and conceptual coverage of large language model (LLM) chess commentary. This framework decomposes commentary into atomic claims and uses engine-supported tools and expert references to verify correctness. Evaluations show that even advanced models like GPT-5.4 hallucinate incorrect sub-claims 22.0% of the time without tool augmentation, while smaller open-weight models exceed 40%. While tool augmentation significantly improves factual correctness and move-quality assessment, the models still struggle to cover expert strategic and tactical ideas comprehensively. AI

IMPACT Highlights the ongoing challenge of factual accuracy in LLM outputs, even in specialized domains, and the need for robust evaluation tools.

RANK_REASON The cluster contains an academic paper detailing a new evaluation framework and benchmark for LLM commentary. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• ACT-Eval
• alphaXiv
• arXiv
• CatalyzeX
• Connected Papers
• CORE Recommender
• DagsHub
• Gotit.pub
• GPT-5.4
• Hugging Face
• Litmaps
• ScienceCast
• scite Smart Citations

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →