StateX framework boosts RNN recall by expanding model states post-training

Researchers have developed StateX, a post-training framework designed to improve the recall capabilities of recurrent neural networks (RNNs). This method efficiently expands the states of pre-trained RNNs, such as linear attention and state-space models, without significantly increasing model parameters. Experiments show StateX enhances recall and in-context learning performance in models up to 1.3 billion parameters, without compromising other functionalities. AI

影响 Enhances recall for RNNs, potentially improving performance on tasks requiring long-context understanding.

排序理由 This is a research paper introducing a new framework for improving RNN performance.

在 arXiv cs.CL 阅读 →

• arXiv
• linear attention
• RNN
• state-space models
• StateX

• 论文
• 模型发布

AI 生成摘要 · Google Gemini · 来自 1 个来源。 我们如何撰写摘要 →