Chai et al. 2024

LLM tokenizers punish random character deletion, increasing costs

An AI sysadmin discovered that randomly deleting characters from LLM prompts to save on token costs actually increases the token count. This occurs because tokenizers, like Byte Pair Encoding (BPE) and SentencePiece, ar…