English(EN) Rounding in Preconditioner Space: Redesigning 4-bit AdamW Optimizer-State Quantization

新的4位量化方法提高了AdamW优化器性能

作者 PulseAugur 编辑部 · [1 个来源] · 2026-10-09 04:00

研究人员开发了量化AdamW算法中优化器状态的新方法，特别关注4位量化。这些技术，ZIP-SR和ZE-EDEN，旨在减少存储并最小化量化引入的误差，这会影响后续的自适应更新。在1.3亿到27亿参数的生成式预训练Transformer和Llama风格模型上的实验表明，这些方法与标准的32位AdamW相比，显著减小了性能差距，其中一种方法将验证损失差距降低了70%。 AI

影响这些量化技术可以通过减少内存需求，在更少的硬件上训练更大的模型。

排序理由该集群包含一篇详细介绍优化AI模型训练的新颖方法的学术论文。[lever_c_demoted from research: ic=1 ai=1.0]

在 arXiv cs.LG 阅读 →

AI 生成摘要 · Google Gemini · 来自 1 个来源。我们如何撰写摘要 →

本文如何被排名

Signal score: 0 / 100; Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket: Tool; 该集群包含一篇详细介绍优化AI模型训练的新颖方法的学术论文。[lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration: Single-source cluster; Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics: paper, infra; Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance: High; Clearly on-topic for AI-industry coverage.
Story freshness: 2 days old; Coverage has settled into its steady-state source set.

完整方法见我们的编辑标准。

报道来源 [1]

arXiv cs.LG TIER_1 English(EN) · Hanyang Li, Shao Tang, Daniel Thomas Braithwaite, Gregory Dexter, Leonardo Neves, Aman Gupta, Hiroto Udagawa, Abhishek Shivanna, Daniel Silva, Rohan Ramanath · 2026-10-09 04:00

预条件器空间中的舍入：重新设计4位AdamW优化器状态量化

arXiv:2610.12444v1 Announce Type: new Abstract: Quantizing AdamW's optimizer states reduces persistent storage, but quantization errors propagate through the moment recurrences and perturb subsequent adaptive updates. We redesign 4-bit optimizer-state quantization for AdamW from …

本文如何被排名

报道来源 [1]

预条件器空间中的舍入：重新设计4位AdamW优化器状态量化

相关实体

相关话题