LoRA adapters internalize documents for closed-book QA, outperforming RAG

Researchers have developed a method to internalize documents directly into the weights of a 4-bit Gemma-4-e4b model using LoRA adapters. This approach allows the model to answer questions about a corpus in a closed-book manner, without needing retrieval or exceeding a context window. The study found that data quality, specifically a single curation pass that shortened answers and removed trivia, significantly improved closed-book accuracy from 57.7% to 85.7%. This internalized adapter also demonstrated lower latency and superior performance compared to a BM25-RAG pipeline. AI

IMPACT This research suggests data quality and efficient fine-tuning methods can significantly boost LLM performance in closed-book QA scenarios, potentially reducing reliance on retrieval systems.

RANK_REASON Academic paper detailing a novel method for LLM training. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• alphaXiv
• arXiv
• BM25-RAG
• CatalyzeX
• DagsHub
• Gemma 4 E4B
• Gotit.pub
• Hugging Face
• Joan Figuerola Hurtado
• LoRA
• ScienceCast

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →