SCBench

GateSight optimizes LLM memory by predicting KV cache utility

Researchers have developed GateSight, a novel method to optimize the KV cache in hybrid Large Language Models (LLMs). GateSight uses lightweight MLPs to predict the future utility of KV entries based on signals from Gat…