Calibrated LLM-as-judge: how I made my LLM give honest 4/10 scores instead of always-an-8
A developer created a system to generate ad scripts, where the LLM initially assigned overly high scores to the generated hooks. To address this, the developer implemented a three-layer approach within the system prompt. This involved providing a calibrated scoring rubric with clear definitions for each score, including worked examples, and enforcing structured JSON output to ensure the LLM adhered to the scoring guidelines, resulting in more realistic score distributions. AI
IMPACT Provides a practical method for improving LLM evaluation accuracy without fine-tuning, enabling more reliable AI-generated content assessment.