Researchers have introduced XM-Sparse, a novel system designed to enhance the efficiency of serving large language models (LLMs) with sparse attention mechanisms. The system addresses the bottleneck of GPU High Bandwidth Memory (HBM) capacity by offloading KV cache and K-index data to scale-up memory pools. XM-Sparse employs tailored I/O mechanisms for each cache type, including TMA-based retrieval for KV entries and SM-efficient DMA prefetching for K-index accesses. Experiments show XM-Sparse significantly boosts decode throughput compared to existing PCIe-based systems, achieving up to 4.98x higher performance on models like DeepSeek V3.2 and GLM 5.3 across various context lengths. AI
IMPACT Enhances LLM serving efficiency, potentially enabling higher concurrency and longer context handling for production models.
RANK_REASON The cluster describes a new research paper detailing a novel system for LLM serving. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →