SciNet: Evaluating AI Agents in Relation-Aware Scientific Literature Retrieval
Researchers have introduced SciNet, a novel dataset designed to improve AI agents' ability to understand relational networks within scientific literature. Current retrieval agents often fail to grasp connections between papers, leading to fragmented knowledge and misinterpretations. SciNet, which includes 269 million papers across seven disciplines and 8,940 tasks, evaluates agents on their ability to understand ego-centric, pairwise, and path-wise relationships. Evaluations showed existing agents performed poorly on these relation-aware tasks, but agents trained with SciNet demonstrated a 25.3% improvement in literature review quality. AI
IMPACT Enhances AI's ability to navigate complex scientific literature, potentially accelerating research discovery and synthesis.