PulseAugur
EN
LIVE 07:46:55

Sparse Delta Memory architecture boosts RNN long-context recall

By PulseAugur Editorial · [2 sources] ·

Researchers have introduced Sparse Delta Memory (SDM), a novel architecture designed to enhance the long-context recall capabilities of linear Recurrent Neural Networks (RNNs). Unlike traditional linear attention models that struggle with long-range dependencies due to fixed state sizes, SDM employs a sparse addressing scheme to manage a large explicit memory. This approach allows for significantly higher state capacity without a proportional increase in computational cost, leading to improved performance on in-context learning and long-context retrieval tasks. Furthermore, by treating the initial state of the SDM memory as a parametric memory, the model demonstrates enhanced performance on common-knowledge and reasoning benchmarks. AI

IMPACT Enhances long-context recall in RNNs, potentially improving performance on tasks requiring extensive memory.

RANK_REASON This is a research paper detailing a new architecture for RNNs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

AI-generated summary · Google Gemini · from 2 sources. How we write summaries →

Sparse Delta Memory architecture boosts RNN long-context recall

COVERAGE [2]

  1. arXiv cs.LG TIER_1 English(EN) · Lo\"ic Cabannes, Pierre-Emmanuel Mazar\'e, Gergely Szilvasy, Matthijs Douze, Maria Lomeli, Ilze Amanda Auzina, Justin Carpentier, Gabriel Synnaeve, Herv\'e J\'egou ·

    Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity

    arXiv:2607.07386v1 Announce Type: new Abstract: Linear attention models allow a fixed state size and a fixed amount of compute per token. However, due to their limited state size, linear attention models fall behind in long-context recall compared to softmax-attention-based trans…

  2. arXiv cs.LG TIER_1 English(EN) · Hervé Jégou ·

    Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity

    Linear attention models allow a fixed state size and a fixed amount of compute per token. However, due to their limited state size, linear attention models fall behind in long-context recall compared to softmax-attention-based transformer architectures. Increasing the state size …

RELATED ENTITIES

RELATED TOPICS