SinkState

SinkState optimizes hybrid LLM serving with adaptive low-rank states

Researchers have developed SinkState, a novel method for optimizing the serving of hybrid Large Language Models (LLMs). This technique addresses the inefficiency of current systems by storing recurrent states in an adap…