Byte-Pair Encoding Formalized: Infinite Tokenization Changes Found in Major LLMs

Researchers have formalized byte-pair encoding (BPE), a common tokenization method used in large language models, to understand how prepending text can alter tokenization. They developed a method using 'radius certificates' to efficiently determine if this alteration is infinite or finite for any given dictionary. This analysis reveals that models like GPT-2, cl100k, o200k, and Llama 3 have infinite origin radii from their initial merge stages, indicating a significant potential for tokenization changes with prepended text. AI

IMPACT Formalizes tokenization changes in LLMs, potentially impacting prompt engineering and model behavior.

RANK_REASON Academic paper detailing a formalization of byte-pair encoding and its implications for LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• Berglund
• byte-pair encoding
• cl100k
• Dalirrooyfard
• GPT-2
• Llama 3
• o200k
• van der Merwe

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →