Token prices are rapidly decreasing, which is expected to lessen the financial burden on users of less common languages. Advances in multilingual and synthetic data are improving model quality. However, a significant gap in tokenization for these languages will persist until AI labs intentionally develop tokenizers that better support them. AI
IMPACT AI tokenization costs are decreasing, but dedicated efforts are still needed to improve support for less common languages.
RANK_REASON The item discusses an observation about AI tokenization costs and multilingual support, rather than a specific event or release.
Read on Mastodon — mastodon.social →
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →