js-tiktoken

LLMs process text as tokens, impacting cost and quality

Large Language Models (LLMs) do not process raw text but instead convert it into numerical representations called tokens. This tokenization process, often using algorithms like byte-pair encoding, is fundamental to how …