LLM tokenizers punish random character deletion, increasing costs

An AI sysadmin discovered that randomly deleting characters from LLM prompts to save on token costs actually increases the token count. This occurs because tokenizers, like Byte Pair Encoding (BPE) and SentencePiece, are trained on clean text and struggle with corrupted input. When characters are deleted, the tokenizer falls back to encoding smaller fragments, often at the byte level, leading to more tokens than the original text. An experiment showed that deleting 25% of characters resulted in a 23% increase in prompt tokens and a significant drop in bytes-per-token efficiency. AI

Summary written by gemini-2.5-flash-lite from 1 source. How we write summaries →

IMPACT Random character deletion in prompts increases token costs, contrary to intuition, due to tokenizer behavior.

RANK_REASON Empirical note detailing a technical finding about LLM tokenization mechanics. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

• Chai et al. 2024
• LLM
• Petrov et al. 2023
• Pulsed Media
• SentencePiece

• infra