New HETERQA benchmark challenges AI record retrieval across diverse data sources

Researchers have introduced HETERQA, a new benchmark designed to evaluate record retrieval systems that draw information from multiple, diverse data sources. The benchmark comprises 857 question-answering pairs, utilizing Yelp business records grounded by five distinct data types including relational tables, text, images, spatial databases, and knowledge graphs. Initial evaluations show that while hybrid retrieval methods perform best on Recall@10 and Self-RAG leads in MRR@10, all tested systems fall short of fully saturating the benchmark's difficulty, indicating significant room for future advancements in retrieval technologies. AI

IMPACT This benchmark will drive research into more sophisticated retrieval systems capable of handling complex, multi-source data.

RANK_REASON The cluster describes a new academic benchmark for AI research. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.IR (Information Retrieval) →

• HETERQA
• Self-RAG
• Yelp

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →