LLM evaluation metrics show stark differences in detecting AI fabrications

A recent experiment comparing two popular LLM-as-judge faithfulness metrics, Ragas and DeepEval, revealed significant discrepancies in their ability to detect fabricated information. While both metrics were applied to the same fabricated RAG system output using GPT-4o, Ragas scored the fabrication as 0.0, whereas DeepEval consistently scored it as 1.0, even praising the output for accuracy. The experiment highlighted that LLM judges excel at detecting meaning inversions but struggle with omissions, such as missing critical information like medication dosages, where deterministic checks proved more reliable. The author proposes a hybrid approach combining LLM judges with deterministic checks for robust AI system evaluation. AI

IMPACT Highlights critical limitations in current LLM evaluation metrics, suggesting a need for hybrid approaches to ensure AI system trustworthiness.

RANK_REASON The item details a controlled experiment comparing two LLM evaluation metrics, presenting findings and a proposed solution. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

• DeepEval
• GPT-4o
• OpenGATE
• Ragas

• product
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →