PulseAugur
EN
LIVE 18:49:30

Cloudflare categorizes AI bots, blocking agents and training bots from ad pages

Cloudflare has introduced a new system for categorizing AI bots into three types: search, agent, and training. Starting September 15, new domains will automatically block agent and training bots from accessing pages with advertisements. This change may present challenges for AI agents that scrape live web pages, potentially returning a "verify you're human" challenge instead of the intended content. AI

IMPACT This change could impact how AI agents access and process web content, potentially requiring developers to adjust their scraping strategies and identify bots more transparently.

RANK_REASON Cloudflare's announcement of a new bot categorization system and default blocking policy for certain AI bots on ad-enabled pages.

Read on dev.to — LLM tag →

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →

Cloudflare categorizes AI bots, blocking agents and training bots from ad pages

How we ranked this

Signal score
31 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Tool
Cloudflare's announcement of a new bot categorization system and default blocking policy for certain AI bots on ad-enabled pages.
Source corroboration
Single-source cluster
Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics
product, policy
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
Breaking (< 6h)
Fresh story with cross-source coverage still developing. Ranking may shift as more sources report.

Full methodology in our editorial standards.

COVERAGE [1]

  1. dev.to — LLM tag TIER_1 English(EN) · amay ·

    cloudflare now sorts ai bots into 3 buckets: search, agent, training. since sept 15, new domains block agent + training bots by default on pages with ads.

    <p>if you build agents that fetch live pages:</p> <p>you can get a challenge page instead of the real content<br /> the code often doesn’t error, it just stores “verify you’re human” and your llm runs with that<br /> so check the content looks right, not just the status code<br /…