Do Neural Retrievers Prefer Certain Documents? Evidence of Learned Relevance Priors
A new research paper reveals that supervised neural retrievers, commonly used in information retrieval, develop an implicit bias towards certain document types. This bias, termed a "relevance prior," is learned from the annotation process itself, which often favors comprehensive, mainstream topics over niche or technical content. Consequently, documents with lower prior are systematically harder to retrieve, even if they are genuinely relevant, creating a findability gap. The study suggests this is a structural limitation of supervised retrieval, where models internalize preferences from their training data. AI
IMPACT This research highlights a potential bias in AI-powered search systems, suggesting that the way data is annotated can lead to models overlooking niche or technical information.