研究人员推出MuonSSM，一个旨在提高状态空间模型（SSM）在序列建模任务中的稳定性和性能的新框架。通过关注内存更新的几何形状而非循环转移矩阵的条件，MuonSSM旨在克服长期序列中的不稳定性与内存退化等问题。该框架包含一个基于动量的路径和一个牛顿-舒尔茨变换，理论上改善了梯度传播和谱条件。在各种基准测试中的实验结果表明，当MuonSSM集成到不同的SSM架构中时，在准确性、鲁棒性和长上下文性能方面取得了持续的提升。 AI
影响 这项研究可能带来更稳定、更准确的长上下文序列建模，惠及自然语言处理、视觉和时间序列分析等领域的应用。
排序理由 该集群包含一篇详细介绍序列建模新模型/框架的学术论文。
AI 生成摘要 · Google Gemini · 来自 2 个来源。 我们如何撰写摘要 →
arXiv:2606.30461v1 Announce Type: new Abstract: State space models (SSMs) have emerged as efficient linear-time alternatives to attention for long-sequence modeling. However, existing SSMs often suffer from instability and memory degradation over extended horizons due to poorly c…
State space models (SSMs) have emerged as efficient linear-time alternatives to attention for long-sequence modeling. However, existing SSMs often suffer from instability and memory degradation over extended horizons due to poorly conditioned first-order updates and unbalanced up…