English(EN) PUFFER: Incremental Fuzzy Deduplication for Continuously Evolving Corpora

新的PUFFER管道加速了海量LLM训练数据的去重

作者 PulseAugur 编辑部 · [1 个来源] · 2026-09-01 04:00

研究人员开发了PUFFER，一种用于大规模、持续演进的语言模型训练语料库的增量模糊去重新管道。PUFFER利用不可变的、带数据集标签的内存映射段进行高效的历史成员资格检查，并采用分层压缩策略来管理筛选扇出。与传统方法相比，这种方法显著降低了维护成本和内存需求，能够更快地摄取数据并实现数据集范围的数据撤回。 AI

影响这种新的数据去重方法可以显著提高训练大型语言模型的效率和可扩展性。

排序理由详细介绍数据处理新技术的学术论文。[lever_c_demoted from research: ic=1 ai=1.0]

在 arXiv cs.AI 阅读 →

AI 生成摘要 · Google Gemini · 来自 1 个来源。我们如何撰写摘要 →

本文如何被排名

Signal score: 25 / 100; Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket: Tool; 详细介绍数据处理新技术的学术论文。[lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration: Single-source cluster; Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics: infra, paper; Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance: High; Clearly on-topic for AI-industry coverage.
Story freshness: Breaking (< 6h); Fresh story with cross-source coverage still developing. Ranking may shift as more sources report.

完整方法见我们的编辑标准。

报道来源 [1]

arXiv cs.AI TIER_1 English(EN) · Xiao Yang, Erik Edward Aldape, Beren Millidge · 2026-09-01 04:00

PUFFER：用于持续演进语料库的增量模糊去重

arXiv:2608.28622v1 Announce Type: cross Abstract: Large language model training corpora grow through successive, often redundant releases, so each release must be deduplicated against both itself and the accumulated history. At trillion-token scale, this requires incremental inge…

本文如何被排名

报道来源 [1]

PUFFER：用于持续演进语料库的增量模糊去重

相关实体

相关话题