English(EN) MUON-RT Reusing the Teacher's Trajectory for Momentum-Free Muon for LLM Self-Distillation

新的 LLM 自蒸馏方法减少内存占用

作者 PulseAugur 编辑部 · [1 个来源] · 2026-11-02 05:00

研究人员开发了 MUON-RT，一种新颖的大型语言模型自蒸馏方法，通过重用教师-学生位移而不是单独的动量缓冲区，显著减少了内存使用。这种方法在新的 arXiv 论文中有所详述，它编码了应用更新的指数加权历史并平滑噪声。实验表明，MUON-RT 在使用量显著减少的持久内存的情况下，提高了准确性，优于标准的 AdamW 和基于动量的 Muon 方法。 AI

影响该方法可以实现更高效的大型语言模型训练，降低硬件要求，并可能加速开发周期。

排序理由该集群包含一篇详细介绍 LLM 自蒸馏新方法的学术论文。[lever_c_demoted from research: ic=1 ai=1.0]

在 arXiv stat.ML 阅读 →

AI 生成摘要 · Google Gemini · 来自 1 个来源。我们如何撰写摘要 →

本文如何被排名

Signal score: 1 / 100; Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket: Tool; 该集群包含一篇详细介绍 LLM 自蒸馏新方法的学术论文。[lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration: Single-source cluster; Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics: paper, infra; Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance: High; Clearly on-topic for AI-industry coverage.
Story freshness: 1 days old; Coverage has settled into its steady-state source set.

完整方法见我们的编辑标准。

报道来源 [1]

arXiv stat.ML TIER_1 English(EN) · Aryan Das, Vinay Kumar Verma, Aakansha Mishra · 2026-11-02 05:00

MUON-RT Reusing the Teacher's Trajectory for Momentum-Free Muon for LLM Self-Distillation

arXiv:2610.41104v1 Announce Type: cross Abstract: Self-distillation methods for large language models with an exponential moving average (EMA) teacher already retain a history of student updates. Muon nevertheless stores a separate momentum buffer, increasing persistent training …

本文如何被排名

报道来源 [1]

MUON-RT Reusing the Teacher's Trajectory for Momentum-Free Muon for LLM Self-Distillation

相关实体

相关话题