English(EN) Simulating Training Spikes of Deep Neural Networks by Layer-wise Random Feature Models

新模型模拟LLM训练中的损失尖峰

作者 PulseAugur 编辑部 · [1 个来源] · 2026-10-12 04:00

研究人员开发了一个理论框架和模拟算法来模拟深度神经网络（DNN）的训练动态，特别是解决了在大语言模型（LLM）预训练期间观察到的间歇性损失尖峰现象。他们的工作表明，具有重尾、幂律分布输入的简单二次模型可以复制这些尖峰。在涉及已知会在Transformer LM中引起尖峰的Adam超参数配置的案例研究中，他们的模拟成功地从给定检查点预测了尖峰行为，并且他们推导出了一个经验上提高了性能的$eta_2$调度。 AI

影响提供了一个理论框架来理解和潜在地缓解LLM预训练期间的损失尖峰。

排序理由该集群包含一篇研究论文，详细介绍了用于模拟DNN训练动态的新理论框架和模拟算法。[lever_c_demoted from research: ic=1 ai=1.0]

在 arXiv cs.LG 阅读 →

AI 生成摘要 · Google Gemini · 来自 1 个来源。我们如何撰写摘要 →

本文如何被排名

Signal score: 12 / 100; Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket: Tool; 该集群包含一篇研究论文，详细介绍了用于模拟DNN训练动态的新理论框架和模拟算法。[lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration: Single-source cluster; Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics: paper, infra; Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance: High; Clearly on-topic for AI-industry coverage.
Story freshness: Breaking (< 6h); Fresh story with cross-source coverage still developing. Ranking may shift as more sources report.

完整方法见我们的编辑标准。

报道来源 [1]

arXiv cs.LG TIER_1 English(EN) · Ran Tian, Courtney Paquette · 2026-10-12 04:00

通过逐层随机特征模型模拟深度神经网络的训练尖峰

arXiv:2610.13524v1 Announce Type: new Abstract: Intermittent ``spikes'' in loss, during the pre-training of Large Language Models (LLMs), remains a mysterious phenomenon both theoretically and empirically. In this paper, we show that simple quadratic models with heavy-tailed, pow…

本文如何被排名

报道来源 [1]

通过逐层随机特征模型模拟深度神经网络的训练尖峰

相关实体

相关话题