Last But Not Least: Boundary Attention CalibratiON for Multimodal KV Cache Compression
Two new research papers propose novel methods for compressing KV caches in large language models to improve inference efficiency. The first paper, PolyKV, introduces a layer-wise optimization framework that applies different compression policies and budgets to transformer layers based on their specific roles. The second paper, BACON, focuses on multimodal LLMs and calibrates attention mechanisms to better retain critical visual information under aggressive compression. AI
IMPACT These methods aim to reduce memory costs and latency in LLM inference, potentially enabling longer context windows and more efficient deployment of multimodal models.