Cloudflare is introducing new policies to address the imbalance caused by generative AI consuming web content without compensation. Starting September 15, 2026, Cloudflare will block AI crawlers from accessing advertising-supported pages by default, while allowing traditional search engine crawlers. The company is also evolving its 'Pay Per Crawl' model to 'Pay Per Use,' enabling publishers to be compensated when their content is used in AI results or by AI agents. This initiative aims to create a more equitable ecosystem by providing tools for content owners to manage permissions, attribution, and payment, thereby supporting publishers who rely on advertising and subscriptions. AI
IMPACT This policy shift could force AI companies to re-evaluate their data acquisition strategies and potentially increase costs, while empowering publishers to monetize their content in the age of generative AI.
RANK_REASON Cloudflare, a major infrastructure provider, is implementing a significant policy change affecting the AI industry's access to web content and establishing a new economic model for publishers.
Read on Mastodon — fosstodon.org →
- AI companies
- Cloudflare
- Google Search
- Matthew Prince
- Pay Per Crawl
- Pay Per Use
- September 15, 2026
- You.com
- AI agents
- AI Overviews
- Gemini Apps
- Googlebot
- Vertex API
AI-generated summary · Google Gemini · from 10 sources. How we write summaries →
