mohit modi

LLM Tokenization Explained Through Common Response Errors

This article explains the concept of tokenization in large language models by examining a common 'error' found in their responses. The author, Mohit Modi, uses this specific issue as a pedagogical tool to illustrate how…