LLMs struggle with specific company data, but retrieval boosts accuracy to 90%

A benchmark tested six large language models (LLMs) on 387 specific construction-related questions, evaluating their performance both independently and with retrieval-augmented generation (RAG). The results showed a dramatic improvement with retrieval, with models averaging a 19% accuracy alone and a 90% accuracy when provided with relevant source documents. The study highlighted that access to specific company data was far more critical than the choice of LLM, and noted that models often hallucinated industry-standard answers when company-specific policies differed, without indicating they were guessing. AI

IMPACT Highlights the critical need for retrieval-augmented generation (RAG) systems to bridge the gap between general LLM knowledge and specific enterprise data.

RANK_REASON Benchmark study evaluating LLM performance on domain-specific data. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

• Claude Opus 5.5
• DeepSeek-V4 Flash
• Gemini 2.5-Flash
• GPT-6.1 Sol
• Llama 4 Maverick
• Mistral Large 3

• product
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →