New OSCAR method slashes LLM KV cache size by 8x, boosts speed

Researchers have developed OSCAR, a novel method for quantizing KV caches to 2-bit precision, significantly reducing memory usage and improving inference speed for large language models. This technique aligns the quantization basis with attention mechanisms, preserving accuracy even with extremely compressed caches. OSCAR has been integrated into serving engines like SGLang and vLLM, demonstrating substantial gains in throughput and decoding speed across various benchmarks and model sizes. AI

IMPACT Enables more efficient deployment of large language models by reducing memory footprint and increasing inference speed.

RANK_REASON The item describes a novel method for KV cache quantization presented in an arXiv paper. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• GLM-4.7
• KV cache
• OSCAR
• Qwen3-32B
• Qwen3-4B-Thinking-2507
• Qwen3-8B
• RULER-NIAH
• SGLang
• vLLM
• Zhou Zhongzhu

• infra
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →