New BitDecoding-FP4 method boosts LLM serving throughput with FP4 KV cache

Researchers have developed BitDecoding-FP4, a novel algorithm-system co-design for efficient and accurate native FP4 KV cache decoding in large language models. This approach addresses accuracy drops and throughput degradation associated with existing FP4 KV systems by introducing techniques like Dual-Residual Query Quantization and Underflow-Aware Probability Scaling. The system also optimizes decoding throughput by removing the softmax bottleneck and fusing operations on paged KV caches. Benchmarks on NVIDIA B300 with Qwen3 models show BitDecoding-FP4 achieves accuracy close to BF16 KV while delivering 2.4-3.0x higher serving throughput. AI

IMPACT This research could significantly reduce the computational cost and increase the speed of serving large language models, making them more accessible and efficient.

RANK_REASON This is a research paper detailing a new algorithm and system co-design for improving LLM inference efficiency. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• bfloat16
• BitDecoding-FP4
• Dual-Residual Query Quantization
• Hadamard rotation
• KV cache
• MXFP4
• NVFP4
• NVIDIA B300
• Outlier-Aware Key Quantization
• Qwen3-30B-A3B
• Qwen3-4B
• Tensor Cores
• Underflow-Aware Probability Scaling

• infra
• paper

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →