New EncBank method boosts LLM efficiency with reusable encoder memory

Researchers have developed EncBank, a novel method for improving Large Language Model (LLM) query efficiency by treating lower layers of a pretrained LLM as a reusable document encoder. This approach compactly stores the outputs of these lower layers for an adapted upper-layer reader, reducing redundant encoding for shared documents. EncBank utilizes a self-distilled suffix adapter that functions across different storage precisions without requiring retraining. Testing on three Qwen backbones demonstrated that 4-bit storage maintains benchmark aggregates within one score point of native-precision EncBank, while also reducing persistent GPU storage by 28.1% in a Qwen3-8B workload. AI

IMPACT This method could significantly reduce computational costs and improve response times for LLMs handling repetitive document queries.

RANK_REASON The cluster contains a research paper detailing a new method for LLM query efficiency. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• EncBank
• Hugging Face
• LLM
• Qwen
• Qwen3.8-27B
• Qwen3_8B
• Terminal-Bench 2.1

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →