New method offloads LLM KV cache to RAM for long context and persistent memory

A new technique has been developed to address memory limitations in local large language models, specifically for handling long contexts and maintaining state across restarts. This method involves offloading the model's KV cache, which stores computed internal states, from VRAM to CPU RAM and disk. A small index in VRAM is used to retrieve relevant KV chunks when needed, allowing models to access contexts up to 800,000 tokens while keeping VRAM usage stable. The system also enables models to resume from their stored state after a process restart, effectively acting as a persistent memory. AI

IMPACT Enables local LLMs to handle significantly longer contexts and retain memory across sessions, potentially improving RAG performance and user experience.

RANK_REASON This is a technical research proof-of-concept detailing a novel method for managing LLM memory, not a commercial release or a widely adopted product. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

• LLM
• Mamba
• MiniCPM-1B
• Qwen2.5
• Qwen2.5-7B-1M
• RTX 5060

• infra
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →