A new study published on arXiv investigates how major AI providers like Anthropic, OpenAI, and Google Vertex AI account for input tokens when processing source code. The research compares the token usage for raw source code text against a method of rendering code as images for vision-language models. Results show significant token reductions when code is presented as images, with aggregate reductions ranging from 75.8% to 86.5% across providers. However, the study highlights substantial differences in how each provider's API handles image-based code, with Gemini showing higher token counts at smaller code lengths compared to text, while Anthropic and OpenAI consistently offer lower counts. AI
IMPACT This research highlights potential cost efficiencies and system design considerations for developers using AI models to process source code, particularly concerning tokenization strategies.
RANK_REASON Academic paper analyzing AI model input accounting. [lever_c_demoted from research: ic=1 ai=1.0]
- Anthropic
- arXiv
- cs.CV
- Gemini
- Google Vertex AI
- OpenAI
- Pixels for Programs?
- Source Code as Text and Images
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
