BellowsKV

BellowsKV method drastically cuts LLM KV cache size with minimal accuracy loss

Researchers have developed BellowsKV, a novel method to compress the key-value cache in large language models, which is a significant bottleneck for GPU serving. Unlike previous methods that quantize or evict tokens wit…