New research shows model size scales with data bytes, not tokens, for optimal compute

A new paper explores the impact of token granularity on language model scaling laws. Researchers trained 988 models with varying parameter counts and compression rates to investigate how tokenization affects compute efficiency. The study found that model parameters should scale proportionally to data size in bytes, not tokens, and that the optimal compression rate decreases with compute, offering guidance for developers. AI

Summary written by gemini-2.5-flash-lite from 1 source. How we write summaries →

IMPACT Provides new insights into optimizing tokenization for compute efficiency in language models.

RANK_REASON Academic paper detailing new findings on tokenization's impact on LLM scaling laws. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• BLT
• BPE
• Kaplan et al.
• Hoffmann et al.
• Tomasz Limisiewicz

• paper
• infra