byte-pair encoding
PulseAugur coverage of byte-pair encoding — every cluster mentioning byte-pair encoding across labs, papers, and developer communities, ranked by signal.
9 day(s) with sentiment data
-
Polish language tokenization limits explored in new research paper
A new research paper examines the limitations of Byte-Pair Encoding (BPE) tokenization when applied to highly inflectional languages like Polish. The study reveals that BPE, which relies on statistical frequency, often …
-
New research disentangles tokenization algorithms for language models
Researchers have introduced two new tokenization algorithms, BottomUpLL and TopDownComp, to disentangle the effects of optimization objectives and search procedures in language model tokenizers. By creating a 2x2 design…
-
First English-Syriac machine translation model developed using Bible corpus
Researchers have developed the first phrase-based Statistical Machine Translation (SMT) model for English-to-Syriac, addressing the challenge of translating an endangered language with complex orthography. The study cre…
-
Sanskrit tokenization penalty higher than English, study finds
A new research paper investigates the tokenization efficiency of Sanskrit compared to English and Hindi when processed by modern language models. The study found that Sanskrit requires significantly more tokens per unit…
-
LLMs process language via numerical tokenization, not understanding
Neural networks, including large language models, do not inherently understand human language; instead, they process text by converting it into numerical representations. This process, known as tokenization, breaks down…
-
New DH-BPE method optimizes language model vocabularies for better compression
Researchers have developed a new method called Dynamic-Programming-Guided Hierarchical BPE (DH-BPE) to optimize vocabulary construction for language models. This approach combines token exposure with hierarchical depend…
-
BuzzASR releases 100+ language-specific speech models, outperforming Whisper
Researchers have developed BuzzASR, a suite of over 100 specialized speech recognition models fine-tuned for individual languages. These models are based on the Whisper architecture and significantly outperform the gene…
-
EvoLen tokenizer uses evolutionary data for DNA language models
Researchers have developed EvoLen, a novel tokenization method for DNA language models (DNALMs) that incorporates evolutionary information. Unlike standard byte-pair encoding (BPE) used in natural language, EvoLen prior…
-
LLM Tokenization Explained: From Text to Numbers for AI Models
Large Language Models (LLMs) require text to be converted into numerical representations called tokens before processing. A token can be a word, part of a word, or punctuation, and each is assigned a unique Token ID. Di…
-
Nested byte-level vocabularies offer deployment flexibility but reduce language model performance
A new research paper explores the concept of nested byte-level vocabularies for language models, demonstrating that while slicing models to operate at different vocabulary sizes is numerically exact and can reduce deplo…
-
LLM Tokenizers Explained: Why Prompts Cost Differently Across Models
Understanding how LLM tokenizers work is crucial for managing costs and predicting model behavior. Tokenizers, often based on Byte-Pair Encoding (BPE), break text into subword units that models process as integers. The …
-
New research explores advanced tokenization for LLMs, improving efficiency and performance · 4 sources tracked
Researchers are developing new methods for tokenizing text in large language models to improve efficiency and performance. One approach, SuTRA, focuses on morphological structure for morphologically rich languages like …
-
Superword tokenization algorithms achieve over 600x speedup
Researchers have developed significantly faster implementations of the BoundlessBPE and SuperBPE algorithms for tokenization, which extend Byte Pair Encoding (BPE) to create "superwords" that combine multiple pretokens.…
-
New method estimates LLM training data composition from vocabularies
Researchers have developed a new method called Quantile-Guided Density Estimation (QGDE) to estimate the composition of hidden training corpora for large language models (LLMs). This technique leverages released tokeniz…
-
Tokenization proven NP-complete and APX-hard, even for binary alphabets
A new research paper demonstrates that tokenization, a fundamental process in natural language processing, is computationally intractable even over bounded alphabets. The study proves that both bottom-up and direct toke…
-
Byte-Pair Encoding Explained: From Compression to NLP
This article explains Byte-Pair Encoding (BPE), a data compression algorithm repurposed for neural machine translation. BPE works by repeatedly identifying and replacing the most frequent adjacent character pairs in a c…
-
Byte-level AI models offer parameter savings and improved text handling
A byte-level model for AI processing offers advantages over traditional tokenized models by directly processing 256 possible byte values, eliminating the need for a vocabulary and its associated embedding table. This ap…
-
Pruned BPE enhances tokenization efficiency by optimizing visible vocabulary
Researchers have introduced Pruned BPE, a novel method to enhance Byte Pair Encoding (BPE) for subword tokenization. Unlike standard BPE, which exposes all learned merge tokens to downstream models, Pruned BPE identifie…
-
New method adapts tokenizers for underrepresented languages
Researchers have developed a method to adapt byte-level BPE tokenizers for underrepresented languages without altering the model's vocabulary size. This approach, called BPE-guided insertion, ensures that new token assi…
-
AI agents could benefit from new token-native storage method
A new research paper proposes "token-native storage" as a more efficient method for storing text data used by AI agents. Instead of traditional UTF-8 encoding, the paper suggests storing text directly as token IDs, whic…