quicktok: a faster tokenizer (exact and byte-identical with tiktoken) [P]
A new C++ tokenizer called quicktok has been developed, offering significant speed improvements over existing solutions. It achieves byte-identical tokenization to tiktoken and is notably faster, running 2-3.6x faster than bpe-openai and 4-11x faster than tiktoken itself. The tokenizer supports various models including cl100k, o200k, GPT-OSS, Llama-3, and Qwen2.5/3, utilizing data structure engineering for enhanced performance. AI
IMPACT Accelerates tokenization workflows, potentially speeding up LLM inference and training processes.