van der Merwe

Byte-Pair Encoding Formalized: Infinite Tokenization Changes Found in Major LLMs

Researchers have formalized byte-pair encoding (BPE), a common tokenization method used in large language models, to understand how prepending text can alter tokenization. They developed a method using 'radius certifica…