Short window attention boosts long-term memory in AI models

Researchers have developed a hybrid architecture combining sliding window attention and xLSTM layers to improve long-term memorization in AI models. Their findings indicate that surprisingly, larger sliding windows can hinder long-context performance by forcing the model to better train the xLSTM's long-term memory. To optimize this, they propose training with stochastically changing window sizes, which significantly enhances performance on both short and long-context tasks. AI

Summary written by gemini-2.5-flash-lite from 1 source. How we write summaries →

IMPACT Introduces a novel architectural approach that could enhance long-context capabilities in future AI models.

RANK_REASON Academic paper detailing a new hybrid architecture for AI models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• xLSTM
• Matthijs Douze

• paper
• model release