Tensor Cache: Eviction-conditioned Associative Memory for Transformers
Researchers have developed a novel memory system called Tensor Cache for Transformers, designed to enhance their ability to handle long contexts. This system combines a sliding-window cache with a second-level fast-weight memory that stores evicted tokens. By compressing and recalling evicted KV pairs efficiently, Tensor Cache aims to improve the trade-off between memory usage and model quality for long-context language modeling and other applications. AI
IMPACT Introduces a method to improve Transformer efficiency for long-context tasks, potentially enabling more capable models.