English(EN) CAST: History-Adaptive Spectral Clipping for LLM Pretraining

新的CAST方法通过自适应裁剪梯度改进LLM预训练

作者 PulseAugur 编辑部 · [1 个来源] · 2026-10-15 04:00

研究人员开发了CAST（Clipping with Adaptive Scales and Thresholds，具有自适应尺度和阈值的裁剪）方法，这是一种用于LLM预训练的新颖方法，它建立在软谱裁剪的基础上。CAST根据梯度的历史和估计的谱范数自适应地调整梯度尺度和裁剪阈值。当应用于在83.89B个token上训练的19.08B参数的Mixture of Experts模型时，CAST在九种其他梯度处理技术上表现出优越的性能，实现了更低的训练和验证损失，并减少了梯度尖峰。该方法还使模型能够以更少的步骤达到最佳基线的最终平滑训练损失。 AI

影响这项新的预训练技术可能导致更高效、更稳定的语言模型训练，从而降低计算成本并加速开发。

排序理由该集群描述了在arXiv论文中提出的一种新的LLM预训练方法。[lever_c_demoted from research: ic=1 ai=1.0]

在 arXiv cs.LG 阅读 →

AI 生成摘要 · Google Gemini · 来自 1 个来源。我们如何撰写摘要 →

本文如何被排名

Signal score: 16 / 100; Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket: Tool; 该集群描述了在arXiv论文中提出的一种新的LLM预训练方法。[lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration: Single-source cluster; Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics: paper, infra; Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance: High; Clearly on-topic for AI-industry coverage.
Story freshness: Breaking (< 6h); Fresh story with cross-source coverage still developing. Ranking may shift as more sources report.

完整方法见我们的编辑标准。

报道来源 [1]

arXiv cs.LG TIER_1 English(EN) · Jiacheng Li, Jianchao Tan, Kefeng Zhang, Yuchen Xie, Xunliang Cai · 2026-10-15 04:00

CAST: LLM 预训练的历史自适应谱裁剪

arXiv:2610.19761v1 Announce Type: new Abstract: Soft spectral clipping (SSC) reduces gradient singular values, but its effect depends on the gradient scale and clipping threshold. We introduce CAST (Clipping with Adaptive Scales and Thresholds), a history-adaptive extension of SS…

本文如何被排名

报道来源 [1]

CAST: LLM 预训练的历史自适应谱裁剪

相关实体

相关话题