Tokenization with Split Trees
Researchers have developed a new subword tokenization method called Tokenization with Split Trees (ToaST). This method optimizes compression by recursively splitting text into binary trees and selecting vocabulary based on an Integer Program relaxation. ToaST has demonstrated an 11% reduction in token counts compared to existing methods like BPE and WordPiece, and improved performance in training 1.5B parameter language models. AI
IMPACT This new tokenization method could lead to more efficient language models by reducing token counts and extending effective context length.