PulseAugur
EN
LIVE 22:41:10

Web page token costs for LLMs: Raw HTML is inefficient, Markdown saves 98%

By PulseAugur Editorial · [1 sources] ·

A study measured the token cost of processing web pages for retrieval-augmented generation (RAG) pipelines and AI agents. The findings indicate that raw HTML pages are significantly token-inefficient, with a median of approximately 128,000 tokens. Converting these pages to LLM-oriented Markdown, after extracting only the main content and performing further cleanup, resulted in a median reduction of 98% fewer tokens compared to raw HTML and 63% fewer than full-page text. The study emphasizes that main-content extraction is the most impactful step in reducing token usage, with savings varying widely depending on the page's structure. AI

IMPACT Optimizing web page processing for LLMs can significantly reduce costs and improve retrieval accuracy in RAG systems.

RANK_REASON The item presents findings from a study measuring token efficiency for web pages processed by LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →

Web page token costs for LLMs: Raw HTML is inefficient, Markdown saves 98%

How we ranked this

Signal score
34 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Tool
The item presents findings from a study measuring token efficiency for web pages processed by LLMs. [lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration
Single-source cluster
Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics
infra, product
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
Breaking (< 6h)
Fresh story with cross-source coverage still developing. Ranking may shift as more sources report.

Full methodology in our editorial standards.

COVERAGE [1]

  1. dev.to — LLM tag TIER_1 English(EN) · Leif Albrand ·

    How many tokens does a web page cost your LLM? We measured 20 real pages

    <p><em>Raw HTML vs. full-page text vs. main-content Markdown — median, range, and the cases where the saving is small.</em></p> <p>If you build RAG pipelines or agents that read the web, every page you fetch ends up as input tokens: in the embedding step, in the context window, o…

RELATED ENTITIES

RELATED TOPICS