Standard BPE

Pruned BPE enhances tokenization efficiency by optimizing visible vocabulary

Researchers have introduced Pruned BPE, a novel method to enhance Byte Pair Encoding (BPE) for subword tokenization. Unlike standard BPE, which exposes all learned merge tokens to downstream models, Pruned BPE identifie…