English(EN) Deriving Neural Scaling Laws from the statistics of natural language

新理论从语言统计预测神经缩放定律

作者 PulseAugur 编辑部 · [1 个来源] · 2026-07-07 04:00

研究人员开发了一种新理论，可以定量预测在自然语言数据集上训练的大型语言模型（尤其是在数据受限的情况下）的神经缩放定律的指数。该理论确定了语言的两个关键统计特性：成对标记相关性的衰减和下一标记条件熵随上下文长度的衰减。推导出的公式没有自由参数，可以根据这些语言统计数据准确预测缩放指数，并已在 TinyStories 和 WikiText 基准上训练的 GPT-2 和 LLaMA 等模型上得到验证。 AI

影响为指导未来 LLM 的开发和资源分配提供了理论框架。

排序理由该集群包含一篇详细介绍神经缩放定律新理论模型的学术论文。[lever_c_demoted from research: ic=1 ai=1.0]

在 arXiv stat.ML 阅读 →

AI 生成摘要 · Google Gemini · 来自 1 个来源。我们如何撰写摘要 →

本文如何被排名

Signal score: 0 / 100; Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket: Tool; 该集群包含一篇详细介绍神经缩放定律新理论模型的学术论文。[lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration: Single-source cluster; Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics: paper, model release; Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance: High; Clearly on-topic for AI-industry coverage.
Story freshness: 64 days old; Aged out of breaking-news scoring windows; ranking reflects the durable signal from the full source set.

完整方法见我们的编辑标准。

报道来源 [1]

arXiv stat.ML TIER_1 English(EN) · Francesco Cagnetta, Allan Ravent\'os, Surya Ganguli, Matthieu Wyart · 2026-07-07 04:00

从自然语言统计中推导神经缩放定律

arXiv:2602.07488v3 Announce Type: replace-cross Abstract: Despite the fact that experimental neural scaling laws have substantially guided empirical progress in large-scale machine learning, no existing theory can quantitatively predict the exponents of these important laws for a…

本文如何被排名

报道来源 [1]

从自然语言统计中推导神经缩放定律

相关实体

相关话题