English(EN) Taming Curvature: Architecture Warm-Up for Stable Transformer Training

新的“架构预热”稳定了Transformer训练

作者 PulseAugur 编辑部 · [2 个来源] · 2026-06-15 14:16

研究人员开发了一种新方法来稳定大型Transformer模型的训练，这类模型通常容易出现不稳定性或发散。该方法称为“架构预热”，通过逐步增加网络深度来管理预条件Hessian，这是与训练不稳定性相关的曲率度量。该技术得到了一个用于Hessian特征值快速在线估计器的支持，已被证明可以在不影响收敛的情况下减少不稳定性。 AI

影响提高了大规模Transformer模型训练的效率和可靠性。

排序理由该集群包含一篇研究论文，详细介绍了一种提高AI模型训练稳定性 novel 的新方法。

在 arXiv cs.LG 阅读 →

AI 生成摘要 · Google Gemini · 来自 2 个来源。我们如何撰写摘要 →

报道来源 [2]

arXiv cs.LG TIER_1 English(EN) · Sameera Ramasinghe, Ajanthan Thalaiyasingam, Hadi Mohaghegh Dolatabadi, Chamin Hewa Koneputugodage, Gil Avraham, Violetta Shevchenko, Yan Zuo, Karol Pajak, Alexander Long · 2026-06-16 04:00

Taming Curvature: Architecture Warm-Up for Stable Transformer Training

arXiv:2606.16768v1 Announce Type: new Abstract: Training billion-parameter Transformers is often brittle, with transient loss spikes and divergence that waste compute. Even though the recently developed Edge of Stability (EoS) theory provides a powerful tool to understand and con…
arXiv cs.LG TIER_1 English(EN) · Alexander Long · 2026-06-15 14:16

Taming Curvature: Architecture Warm-Up for Stable Transformer Training

Training billion-parameter Transformers is often brittle, with transient loss spikes and divergence that waste compute. Even though the recently developed Edge of Stability (EoS) theory provides a powerful tool to understand and control the stability of optimization methods via t…

报道来源 [2]

Taming Curvature: Architecture Warm-Up for Stable Transformer Training

Taming Curvature: Architecture Warm-Up for Stable Transformer Training

相关实体

相关话题