研究人员推出了一种新颖的潜在空间语言模型NCP-ArchPreview，它超越了传统的下一个词元预测。该模型集成了下一概念预测（NCP），使其能够学习和预测跨越多个词元的离散概念。该模型拥有89亿参数，在Dolma-3数据集的5.73万亿词元上进行训练，仅使用了51.3%的训练词元就达到了OLMo-3-7B的预训练损失，并在下游任务上取得了显著的提升，包括在GSM8K上提高了5.99个百分点。 AI
影响 这种新的语言建模方法可能带来更高效的训练和在复杂推理任务上性能的提升。
排序理由 该集群描述了一份技术报告，详细介绍了一种新的语言模型架构及其性能，符合研究类别。
在 Hugging Face Daily Papers 阅读 →
AI 生成摘要 · Google Gemini · 来自 6 个来源。 我们如何撰写摘要 →
完整方法见我们的编辑标准。
arXiv:2609.10715v1 Announce Type: new Abstract: We introduce NCP-ArchPreview, a latent-space language model that pushes autoregressive pretraining beyond standard next-token prediction (NTP). Alongside NTP, the model learns through Next Concept Prediction (NCP) to predict discret…
NCP-ArchPreview is a large latent-space language model that jointly trains next-token and next-concept prediction to improve pretraining efficiency and downstream performance.
Shanghai Artificial Intelligence Laboratory and Shanghai Jiao Tong University's LUMIA Lab open-sourced NCP-ArchPreview, an 8.9B latent-space model that jointly trains next-concept and next-token prediction, matching a 7B baseline loss on ~51.3% of tokens.
Shanghai AI Lab and SJTU have open-sourced NCP-ArchPreview, an 8.9B parameter latent-space language model. The model jointly trains next-concept and next-token prediction, achieving a 7B baseline loss using just 51.3% of the training tokens. This marks the largest demonstration o…
📄 Paper «NCP-ArchPreview» shifts language modeling from next token to next concept prediction in latent space. 177 upvotes on HF suggest strong interest. https:// huggingface.co/papers/2609.107 15 # AI # MachineLearning # Research
📄 'NCP-ArchPreview Technical Report' moves from token prediction to next concept prediction in latent space. With 22 upvotes on Hugging Face, this approach rethinks language model reasoning. Read the report: https:// huggingface.co/papers/2609.107 15 # AI # MachineLearning # Rese…