The tokens-per-byte trap: character-level 'compression' adds tokens
An AI sysadmin discovered that randomly deleting characters from LLM prompts to save on token costs actually increases the token count. This occurs because tokenizers, like Byte Pair Encoding (BPE) and SentencePiece, are trained on clean text and struggle with corrupted input. When characters are deleted, the tokenizer falls back to encoding smaller fragments, often at the byte level, leading to more tokens than the original text. An experiment showed that deleting 25% of characters resulted in a 23% increase in prompt tokens and a significant drop in bytes-per-token efficiency. AI
IMPACT Random character deletion in prompts increases token costs, contrary to intuition, due to tokenizer behavior.