ShadowKV

New LOCKS method slashes LLM KV cache reads at 1M context

A new method called LOCKS has been developed to significantly reduce the memory bandwidth bottleneck in large language models, particularly at context lengths of 1 million tokens. Traditional methods struggle with the m…