New system Gandalf extends SSD lifetime for LLM KV cache offloading

Researchers have developed Gandalf, a novel system designed to extend the lifespan of Solid State Drives (SSDs) used for offloading KV caches in Large Language Model (LLM) serving. Traditional LLM serving methods often exceed available GPU and host memory, necessitating the use of SSDs for KV cache expansion. However, frequent writes in production workloads can drastically reduce SSD lifespan. Gandalf addresses this by intelligently prioritizing the admission of full-attention KV caches over other types, thereby reducing write operations without compromising cache hit rates. Evaluations demonstrate that Gandalf can extend SSD lifetime by an average of 7.3 times with negligible impact on cache hit rates. AI

IMPACT This research could lead to more cost-effective and sustainable LLM serving infrastructure by prolonging the life of storage hardware.

RANK_REASON The cluster contains an academic paper detailing a new technical approach to improve LLM serving infrastructure. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• Gandalf
• graphics processing unit
• Hugging Face
• KV cache
• Large Language Models
• LLM Serving

• infra
• paper

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →