Researchers have introduced MAPLE, a new benchmark designed to evaluate scientific paper retrieval systems. Unlike existing benchmarks that focus on single query-paper relevance, MAPLE assesses a retriever's ability to consistently find a paper based on its motivation, methods, and experimental findings. The benchmark includes 2,095 queries for recent machine learning and natural language processing papers, incorporating both textual and multimodal content. Experiments show a significant performance gap, with the best models struggling to retrieve papers across all aspects, highlighting the need for more comprehensive retrieval methods. AI
IMPACT This benchmark could drive the development of more sophisticated AI systems for scientific literature search and analysis.
RANK_REASON The item describes a new benchmark for evaluating scientific paper retrieval systems, which falls under research. [lever_c_demoted from research: ic=1 ai=1.0]
Read on arXiv cs.IR (Information Retrieval) →
- alphaXiv
- arXiv
- CatalyzeX Code Finder for Papers
- DagsHub
- Gotit.pub
- Hugging Face
- Influence Flower
- MAPLE-Synth
- natural language processing
- OpenReview
- ScienceCast
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
