研究人员推出了一种新颖的优化方法SoftSignum，旨在改进深度学习中的参数异质性处理。该技术通过温度控制的软符号变换来平滑基于符号的更新机制，从而实现从类符号到幅度敏感方法的自适应步长转换。实验（包括LLM预训练）表明，SoftSignum及其矩阵值对应物SoftMuon的性能优于AdamW等现有方法。 AI
影响 引入了一种新的优化方法，可以提高大型语言模型和其他深度学习任务的训练稳定性和收敛性。
排序理由 该集群包含一篇详细介绍深度学习模型新优化技术的论文。
AI 生成摘要 · Google Gemini · 来自 2 个来源。 我们如何撰写摘要 →
arXiv:2605.31371v1 Announce Type: new Abstract: Sign-based and LMO-inspired optimizers have recently attracted substantial attention in deep learning due to their strong performance and low memory footprint. However, their fixed-magnitude updates can hurt terminal convergence: th…
Sign-based and LMO-inspired optimizers have recently attracted substantial attention in deep learning due to their strong performance and low memory footprint. However, their fixed-magnitude updates can hurt terminal convergence: they decouple update mechanisms from gradient magn…