LLM code context: Comments and docstrings are the real token cost, not whitespace

Analysis of the CPython 3.13 standard library reveals that comments and docstrings constitute approximately 34.8% of tokens, making them the most significant portion of code sent to LLMs. In contrast, whitespace, such as indentation and blank lines, accounts for a much smaller percentage, with indentation saving only 2.5% when reduced. This suggests that efforts to reduce token count for LLM context windows should prioritize stripping prose over whitespace, as the latter offers minimal savings. AI

IMPACT Focusing on prose reduction in code can significantly cut LLM token costs for AI coding assistants.

RANK_REASON Analysis of code tokenization for LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

• Claude Haiku 5.5
• Claude Sonnet 5.5
• CPython
• GPT-5.6
• OpenAI
• Python
• tiktoken

• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →