研究人员开发了新的超参数迁移方法，实现了大型神经网络更有效的扩展。一篇论文介绍了一种由动力学平均场理论证实的参数化方法，能够实现从5100万到超过20亿参数的模型之间的可靠超参数迁移。另一项研究量化了超参数迁移，并强调了嵌入层学习率的关键作用，表明最大化其学习率可以显著提高训练稳定性和性能，尤其是在使用AdamW优化器时。 AI
影响 新的参数化和优化技术可以显著降低大规模AI模型训练的成本和复杂性。
排序理由 该集群包含两篇学术论文，详细介绍了关于超参数迁移和模型参数化的新研究。
在 Hugging Face Daily Papers 阅读 →
AI 生成摘要 · Google Gemini · 来自 4 个来源。 我们如何撰写摘要 →
arXiv:2601.20205v3 Announce Type: replace Abstract: Mixture-of-Experts (MoE) layers have emerged as an important tool in scaling up modern neural networks by decoupling total trainable parameters from activated parameters in the forward pass for each token. However, sparse MoEs a…
Hyperparameter transfer allows extrapolating optimal optimization hyperparameters from small to large scales, making it critical for training large language models (LLMs). This is done either by fitting a scaling law to the hyperparameters or by a judicious choice of parameteriza…
arXiv:2605.21486v1 Announce Type: cross Abstract: Hyperparameter transfer allows extrapolating optimal optimization hyperparameters from small to large scales, making it critical for training large language models (LLMs). This is done either by fitting a scaling law to the hyperp…
Hyperparameter transfer allows extrapolating optimal optimization hyperparameters from small to large scales, making it critical for training large language models (LLMs). This is done either by fitting a scaling law to the hyperparameters or by a judicious choice of parameteriza…