LLM judges fail to distinguish human from AI writing, study finds

A study using LLM judges to evaluate human-written text found that the models consistently misidentified AI-generated content as human-written and vice-versa. The judges showed high agreement but low accuracy, often mistaking detail density and perfect structure for human authenticity. While one model family showed improvement with specific prompting, others remained biased, indicating deeper issues beyond prompt-layer fixes. The researchers concluded that LLM judges are unreliable for evaluating human-like writing without rigorous calibration against human consensus. AI

IMPACT LLM judges are unreliable for evaluating writing quality, necessitating human oversight and rigorous calibration for AI-generated content.

RANK_REASON The item describes an experiment and its findings regarding the performance of LLM judges, which constitutes research. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

• Claude
• DeepSeek
• General Language Model
• Qwen

• model release
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →