New CoffeeBench benchmark tests LLM agents for appliance fault diagnosis

A new benchmark called CoffeeBench has been developed to assess the ability of large language model (LLM) agents to diagnose faults in simulated appliances within an Industry 4.0 context. While LLMs struggle to consistently identify faults using only sensor data, the best-performing models can iteratively refine their diagnostic hypotheses. The research highlights that initial diagnostic uncertainty and the LLM's self-assessed reasoning soundness are strong predictors of success, and suggests practical improvements like explicit state tracking and deliberation loops to enhance performance. AI

IMPACT This research introduces a new benchmark for evaluating LLM agents in industrial fault diagnosis, potentially improving automation in Industry 4.0.

RANK_REASON The item is an academic paper detailing a new benchmark and research findings. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• alphaXiv
• arXiv
• CatalyzeX Code Finder for Papers
• CoffeeBench
• Connected Papers
• CORE Recommender
• DagsHub
• Gotit.pub
• Hugging Face
• Industry 4.0
• Influence Flower
• Litmaps
• natural language processing
• Pierre-Henri Murcia
• ScienceCast
• scite Smart Citations

• paper
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →