Cloudflare is implementing new rules starting September 15th to manage AI agent crawlers accessing web content. The company is replacing its single "block-AI-bots" setting with three distinct categories: Search, Agent, and Training. While Search bots will generally be allowed, Agent bots, which act in real-time for users, and Training bots will be blocked by default on ad-supported pages. This change aims to encourage better separation of bot functions and potentially lead to new content licensing models, as publishers may start charging for access to their data. AI
IMPACT This policy shift could significantly impact how AI agents access web data, potentially leading to new content licensing models and reduced data availability for training.
RANK_REASON Policy change by a major web infrastructure provider affecting AI agent access to content. [lever_c_demoted from significant: ic=1 ai=0.7]
Read on Artificial Intelligence News →
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
