New method calibrates LLM judgments for better document reranking evaluation

Researchers have developed a new method called Rubric-Calibrated Preferences (RCP) to improve the evaluation of document reranking systems. Traditional metrics like nDCG struggle with the limitations of human-provided relevance labels. RCP combines relative judgments from LLMs within a query with absolute criteria from a rubric, using Item Response Theory to create a unified scale across queries. This approach, termed RCP-nDCG, generates more accurate and comparable relevance probabilities than standard methods, showing improved correlation with human judgments and better differentiation between reranking systems. AI

IMPACT This new method could lead to more reliable evaluations of AI-powered search and recommendation systems, improving their development and deployment.

RANK_REASON The cluster contains an academic paper detailing a new methodology for evaluating AI systems. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.IR (Information Retrieval) →

• Fabian David Schmidt
• item response theory
• LLM
• NanoBEIR
• nDCG
• NIST
• RCP-nDCG
• Rubric-Calibrated Preferences
• TREC DL

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →