PulseAugur
EN
LIVE 10:55:55

Simon Willison releases ttok 1.0, defaults to GPT-5/GPT-6 tokenizer

By PulseAugur Editorial · [2 sources] ·

Simon Willison has released version 1.0 of his command-line tool, ttok, which counts tokens using OpenAI's tiktoken library. The update includes a fix for a Click warning, updated continuous integration, and a new command to list available models. Willison also noted that the tool now defaults to the GPT-5/GPT-6 tokenizer, based on research suggesting these models share a tokenizer, though OpenAI has not officially confirmed this. AI

IMPACT This update to a token counting tool may improve developer workflows for interacting with newer OpenAI models.

RANK_REASON A tool release and update for an existing utility.

Read on Simon Willison →

AI-generated summary · Google Gemini · from 2 sources. How we write summaries →

Simon Willison releases ttok 1.0, defaults to GPT-5/GPT-6 tokenizer

How we ranked this

Signal score
0 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Tool
A tool release and update for an existing utility.
Source corroboration
2 independent sources
Multiple independent publishers reporting the same story raises confidence that it's real and newsworthy.
Topics
product, model release
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
2 days old
Coverage has settled into its steady-state source set.

Full methodology in our editorial standards.

COVERAGE [2]

  1. Simon Willison TIER_1 (HU) ·

    ttok 1.0

    <p><strong>Release:</strong> <a href="https://github.com/simonw/ttok/releases/tag/1.0">ttok 1.0</a></p> <p>I released <a href="https://simonwillison.net/2026/Oct/8/ttok/">ttok 0.4</a>, ran <code>uv tool upgrade ttok</code>, piped a file into the new version... and realized that i…

  2. Simon Willison TIER_1 (HU) ·

    ttok 0.4

    <p><strong>Release:</strong> <a href="https://github.com/simonw/ttok/releases/tag/0.4">ttok 0.4</a></p> <p><code>ttok</code> is my CLI tool for counting tokens, using OpenAI's open source <a href="https://github.com/openai/tiktoken">tiktoken</a> library.</p> <p>It hasn't been in …

RELATED ENTITIES

RELATED TOPICS