A study on the Proof of Agent (PoA) network examined rubric drift among AI graders by using a fixed set of synthetic answers. Sixteen grading lanes were tested on six pre-written answers, revealing that most score variation stemmed from the answers themselves rather than grader disagreement on clear boundaries. The research found that rubric drift tends to move in one direction and identified specific axes of disagreement, such as whether a cost-scaling mechanism deserves credit if an attacker can sidestep it. This methodology, which uses shared boundary cases to sharpen rubrics in the open, is applicable to other multi-agent evaluation systems like LLM-as-judge pipelines. AI
IMPACT This study offers a methodology for improving the consistency and fairness of AI grading systems, applicable to LLM-as-judge pipelines.
RANK_REASON The item describes a study and methodology for evaluating AI systems, fitting the research bucket. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
