一项名为“范数分离延迟定律”的新理论解释了“grokking”现象，即模型在记忆训练数据后仍能泛化。研究人员证明了grokking是由范数驱动的表征相变，并建立了延迟与权重衰减和学习率等因素之间的数学关系。这项工作将grokking重新定义为范数分离的可预测结果，并提供了一种预测grokking延迟的算法。 AI
影响 为理解和预测模型泛化延迟提供了理论框架，可能实现更高效的训练。
排序理由 该集群包含两篇arXiv论文，对机器学习优化算法和现象进行了理论分析。
AI 生成摘要 · Google Gemini · 来自 2 个来源。 我们如何撰写摘要 →
arXiv:2603.13331v2 Announce Type: replace-cross Abstract: Grokking -- the sudden generalisation that appears long after a model has perfectly memorised its training data -- has been widely observed but lacks a quantitative theory explaining the length of the delay. We show that g…
arXiv:2601.07326v2 Announce Type: replace-cross Abstract: This paper studies the AdamW-style Shampoo optimizer, an effective implementation of classical Shampoo that notably won the external tuning track of the AlgoPerf neural network training algorithm competition. Our analysis …