byte-pair encoding
PulseAugur coverage of byte-pair encoding — every cluster mentioning byte-pair encoding across labs, papers, and developer communities, ranked by signal.
10 day(s) with sentiment data
-
Superword tokenization algorithms achieve over 600x speedup
Researchers have developed significantly faster implementations of the BoundlessBPE and SuperBPE algorithms for tokenization, which extend Byte Pair Encoding (BPE) to create "superwords" that combine multiple pretokens.…
-
New method estimates LLM training data composition from vocabularies
Researchers have developed a new method called Quantile-Guided Density Estimation (QGDE) to estimate the composition of hidden training corpora for large language models (LLMs). This technique leverages released tokeniz…
-
Tokenization proven NP-complete and APX-hard, even for binary alphabets
A new research paper demonstrates that tokenization, a fundamental process in natural language processing, is computationally intractable even over bounded alphabets. The study proves that both bottom-up and direct toke…
-
Byte-Pair Encoding Explained: From Compression to NLP
This article explains Byte-Pair Encoding (BPE), a data compression algorithm repurposed for neural machine translation. BPE works by repeatedly identifying and replacing the most frequent adjacent character pairs in a c…
-
Byte-level AI models offer parameter savings and improved text handling
A byte-level model for AI processing offers advantages over traditional tokenized models by directly processing 256 possible byte values, eliminating the need for a vocabulary and its associated embedding table. This ap…
-
Pruned BPE enhances tokenization efficiency by optimizing visible vocabulary
Researchers have introduced Pruned BPE, a novel method to enhance Byte Pair Encoding (BPE) for subword tokenization. Unlike standard BPE, which exposes all learned merge tokens to downstream models, Pruned BPE identifie…
-
New method adapts tokenizers for underrepresented languages
Researchers have developed a method to adapt byte-level BPE tokenizers for underrepresented languages without altering the model's vocabulary size. This approach, called BPE-guided insertion, ensures that new token assi…
-
AI agents could benefit from new token-native storage method
A new research paper proposes "token-native storage" as a more efficient method for storing text data used by AI agents. Instead of traditional UTF-8 encoding, the paper suggests storing text directly as token IDs, whic…
-
New PCAP-LM representation enables LLMs to analyze network traffic
Researchers have developed PCAP-LM, a novel text representation designed to make network traffic data compatible with large language models (LLMs). This method uses a custom alphabet called PacketGlyphs to semantically …
-
New JOLT method optimizes tokenization, outperforming BPE
Researchers have developed a new method called Joint Optimization for Greedy Longest-Match Tokenization (JOLT) to improve subword vocabulary training. JOLT formulates vocabulary learning as an integer program, ensuring …
-
New BHARATI tokenizers boost efficiency for classical Indian languages
Researchers have developed BHARATI, a new set of tokenizers specifically designed for classical Indian languages like Sanskrit and Tamil. Unlike standard algorithms that struggle with the agglutinative morphology and sa…
-
GPT-2's byte-level BPE tokenization ensures full coverage, preventing out-of-vocabulary issues
The GPT-2 paper introduced a significant advancement in tokenization by utilizing Byte Pair Encoding (BPE) over UTF-8 bytes instead of Unicode code points. This byte-level BPE approach guarantees that no input string, i…
-
Language model fine-tuned to translate space-less Khmer language
A language model was fine-tuned to translate Khmer, a language that lacks spaces between words, using a dataset of 8,000 sentences and a single GPU. The process involved adapting tokenization methods like WordPiece and …
-
New tokenizer method improves cross-lingual homograph handling in LLMs
Researchers have identified a limitation in multilingual language models where shared subword vocabularies can lead to identical surface forms being treated too uniformly across languages, even when their meanings diffe…
-
Understanding GPT: Tokens, Transformers, and Training Explained
This article provides a practical guide to understanding Generative Pre-trained Transformers (GPT), explaining that they are neural language models designed to process and predict sequences of tokens. It details how GPT…
-
New tokenizer and playbook aim to cut LLM token costs
Researchers have developed Thunder-Tok, a new subword tokenizer designed to reduce token counts without sacrificing performance in large language models. This method achieves approximately 25% fertility reduction in Eng…
-
New tokenization method boosts cross-lingual fairness in NLP
Researchers have developed a new tokenization algorithm called Parity-Aware Byte-Pair Encoding (BPE) to address cross-lingual fairness issues in natural language processing. Traditional BPE methods favor dominant langua…
-
BPE vs. Unigram-LM: Tokenization algorithms create distinct vocabularies for chemistry SMILES
A new research paper explores the differences between two common tokenization methods, byte-pair encoding (BPE) and Unigram-LM, when applied to chemical SMILES strings. The study found that these algorithms produce sign…
-
New research tackles LLM alignment, safety, and optimization challenges
Researchers are exploring new methods to improve the alignment and reliability of large language models (LLMs). One study identifies a vulnerability in byte-pair encoding (BPE) tokenization that can be exploited to bypa…
-
Research paper questions LLM pre-training costs for genomics tasks
A new research paper assesses the effectiveness of pre-training large language models (LLMs) for genomics tasks. The study questions whether the significant computational cost of pre-training transformer-based models li…