PulseAugur
EN
LIVE 09:49:04

Sparse Delta Memory boosts linear RNNs for better long-context recall

By PulseAugur Editorial · [4 sources] ·

Researchers have introduced Sparse Delta Memory (SDM), a novel architecture designed to enhance the long-context recall capabilities of linear RNNs. By employing a sparse addressing scheme, SDM significantly increases the hidden state capacity of gated linear RNNs, outperforming traditional transformer architectures on in-context learning and long-context retrieval tasks under similar computational constraints. The architecture extends the Gated DeltaNet by utilizing sparse reads and writes to an explicit memory, and can further improve performance on common-knowledge and reasoning tasks when its initial state is learned as a parametric memory. AI

IMPACT Enhances long-context recall in RNNs, potentially offering an alternative to transformer architectures for specific tasks.

RANK_REASON The cluster describes a new research paper detailing a novel architecture for RNNs.

Read on arXiv cs.LG →

AI-generated summary · Google Gemini · from 4 sources. How we write summaries →

Sparse Delta Memory boosts linear RNNs for better long-context recall

How we ranked this

Signal score
0 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Research
The cluster describes a new research paper detailing a novel architecture for RNNs.
Source corroboration
4 independent sources
Strong cross-source corroboration — multiple independent publishers covered this within the clustering window.
Topics
paper, model release
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
100 days old
Aged out of breaking-news scoring windows; ranking reflects the durable signal from the full source set.

Full methodology in our editorial standards.

COVERAGE [4]

  1. arXiv cs.LG TIER_1 English(EN) · Lo\"ic Cabannes, Pierre-Emmanuel Mazar\'e, Gergely Szilvasy, Matthijs Douze, Maria Lomeli, Ilze Amanda Auzina, Justin Carpentier, Gabriel Synnaeve, Herv\'e J\'egou ·

    Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity

    arXiv:2607.07386v1 Announce Type: new Abstract: Linear attention models allow a fixed state size and a fixed amount of compute per token. However, due to their limited state size, linear attention models fall behind in long-context recall compared to softmax-attention-based trans…

  2. arXiv cs.LG TIER_1 English(EN) · Hervé Jégou ·

    Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity

    Linear attention models allow a fixed state size and a fixed amount of compute per token. However, due to their limited state size, linear attention models fall behind in long-context recall compared to softmax-attention-based transformer architectures. Increasing the state size …

  3. Hugging Face Daily Papers TIER_1 English(EN) ·

    Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity

    Sparse Delta Memory extends gated linear RNNs with sparse addressing to dramatically increase hidden state capacity for improved long-context learning and retrieval while maintaining computational efficiency.

  4. r/LocalLLaMA TIER_1 English(EN) · /u/pmttyji ·

    [Paper] Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity

    <table> <tr><td> <a href="https://www.reddit.com/r/LocalLLaMA/comments/1usdn4c/paper_sparse_delta_memory_scaling_the_state_of/"> <img alt="[Paper] Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity" src="https://preview.redd.it/4y84i8701cch1.jpg?width=140&amp;…

RELATED ENTITIES

RELATED TOPICS