Apple researchers unveil weighted state souping for efficient long-context retrieval

Apple Machine Learning Research has introduced Weighted State Souping, a novel method to enhance the efficiency of long-context retrieval in State Space Models (SSMs). This technique addresses the limitations of existing SSMs and state souping methods, which struggle with long and complex contexts. By learning query-dependent weights for combining cached states, Weighted State Souping significantly improves performance on tasks such as document question answering, retrieval, variable tracking, and tool use. An efficient sparse variant further reduces encoding costs without sacrificing performance, offering a promising approach for language modeling over large collections of information. AI

IMPACT Enhances efficiency for AI agents and models handling large volumes of information.

RANK_REASON The item describes a research paper detailing a new method for improving language model performance. [lever_c_demoted from research: ic=1 ai=1.0]

Read on Apple Machine Learning Research →

• Apple Inc.
• Aryo Lotfi
• Cengiz Pehlevan
• Emmanuel Abbe
• Eran Malach
• Harvard University
• Josh Susskind
• long-context retrieval
• NeurIPS 2026
• Recursive Language Models (RLMs)
• State Space Models (SSMs)
• Weighted State Souping
• William L. Tong
• Workshop on Personalized, Aligned, Long-term Memory for AI Systems (PALM)

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →