LLM-as-Judge Is Three Decisions
Evaluating LLM outputs requires careful consideration of context, the unit of measurement, and the specific dimension being assessed. Simply asking an LLM to provide a numerical score can be misleading if these upstream decisions are not properly defined. The author emphasizes that before prompting an LLM judge, one must first decide on the relevant context, the appropriate unit (e.g., a single turn, an entire conversation, or across multiple sessions), and the specific dimension of quality to be measured, such as accuracy or helpfulness. AI
IMPACT Reframes LLM evaluation from prompt engineering to foundational context and dimension choices, impacting how developers build and assess AI systems.