LLM research synthesis: Wiki vs. Vector RAG comparison shows mixed results

A preregistered comparison evaluated two methods for enabling LLMs to answer questions over a small research corpus: a single-round Vector RAG system and an LLM-compiled markdown wiki. Both systems used the same answer-generating model and were scored by two blinded LLM judges on 13 questions across 24 papers. The wiki performed better at synthesizing information across papers, but its advantage was not statistically significant when both judges' scores were combined. RAG met its prediction for single-fact lookup, though one judge would have refuted it. The wiki was significantly more expensive to query, reversing the prediction that it would be cheaper to run. AI

IMPACT Highlights that LLM-based research synthesis is not a monolithic capability, with systems differing in evidence organization, citation support, cost, and scoring granularity.

RANK_REASON The cluster contains an academic paper detailing a comparison of two LLM-assisted research synthesis methods. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• LLM
• Vector RAG

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →