Can LLM Rerankers Predict Their Own Ranking Performance?
Researchers have developed methods for Large Language Models (LLMs) to predict their own ranking performance without external tools. The study explores both training-free and training-based approaches, examining self-consistency across sampled rankings and direct verbalized confidence. Experiments on TREC Deep Learning datasets indicate that self-consistency is competitive with existing state-of-the-art methods and offers better calibration, while direct verbalized confidence tends to be overconfident. AI
IMPACT This research could improve the efficiency of information retrieval systems by allowing LLMs to self-assess their ranking quality.