PReMISE: Policy Rubrics as Measurement Specifications for LLM Judges
Researchers have developed PReMISE, a framework designed to evaluate the effectiveness of rubrics used by Large Language Model (LLM) judges. The framework treats rubrics as measurement specifications, analyzing their structural adequacy, reliability, preference fit, and adversarial robustness. Findings indicate that no single rubric source is simultaneously reliable, preference-predictive, and robust against exploitation, and PReMISE offers repair operations to improve judge accuracy and reduce the rate of exploitable responses receiving high scores. AI
IMPACT Enhances the reliability and trustworthiness of LLM-based evaluations, crucial for model development and safety.