byte-pair encoding
PulseAugur coverage of byte-pair encoding — every cluster mentioning byte-pair encoding across labs, papers, and developer communities, ranked by signal.
2 天有情绪数据
-
LLM 分词器惩罚随机字符删除，增加成本
一位 AI 系统管理员发现，为了节省 token 成本而从 LLM prompt 中随机删除字符实际上会增加 token 数量。这是因为像字节对编码 (BPE) 和 SentencePiece 这样的分词器是在干净文本上训练的，并且难以处理损坏的输入。当删除字符时，分词器会回退到对更小的片段进行编码，通常是在字节级别，导致 token 数量比原始文本更多。一项实验表明，删除 25% 的字符会导致 prompt token 数量增加 2…
-
新的ConvexTok算法使用凸优化来优化NLP分词
研究人员开发了一种新的分词算法，称为ConvexTok，它使用凸优化来构建分词器。与现有的贪婪方法（如BPE和Unigram）不同，ConvexTok考虑整个词汇表以做出最优决策。该算法在分词指标、语言模型的每字节比特数方面表现出改进，并提供了最优性的认证，在常见的词汇量大小下，其结果接近最优值的1%。
-
新的ToaST分词器将词元数量减少了11%以上
研究人员开发了一种名为基于分裂树的分词（ToaST）的新子词分词方法。该方法通过将文本递归地分裂成二叉树并基于整数规划松弛选择词汇来优化压缩。与BPE和WordPiece等现有方法相比，ToaST在词元数量上减少了11%，并在训练1.5B参数语言模型方面取得了更好的性能。
-
Paper analyzes how data representation impacts Transformer context
A new paper analyzes how different representations of data, such as bytes, characters, or subword tokens, affect the performance of Transformer models. The research introduces 'fragmentation' to explain why smaller unit…
-
New research shows model size scales with data bytes, not tokens, for optimal compute
A new paper explores the impact of token granularity on language model scaling laws. Researchers trained 988 models with varying parameter counts and compression rates to investigate how tokenization affects compute eff…
-
New research boosts LLM edge inference speed and cross-model circuit transfer
Researchers have developed Peek2, a new pretokenizer for Byte-level BPE tokenizers that offers a significant speedup for LLM inference on edge devices. This drop-in replacement increases throughput by up to 2.48x in mic…
-
Interactive guide explains how large language models like ChatGPT are built
A new interactive visual guide, based on Andrej Karpathy's lecture, explains the intricate process of building large language models. It details the journey from collecting vast amounts of internet text to the final sta…