Cloudflare has begun implementing the HTTP 402 Payment Required status code for its Pay-Per-Crawl service, a move that significantly alters how web crawlers interact with websites. Previously, crawlers faced 200 OK for access or 403 Forbidden for restrictions. The new 402 code acts as a quote for a fee, requiring crawlers to re-request with an exact price to gain access, effectively turning access control into a transactional process. A naive implementation of this new code can lead to significantly overspending, as demonstrated by a test where a simple agent spent 10x its budget, while a more sophisticated handler with a price cap and fallback options managed costs effectively and retrieved fewer pages. AI
IMPACT Web crawlers and AI agents will need new logic to handle transactional access, impacting data acquisition costs and strategies.
RANK_REASON This is a new product feature implementation by Cloudflare, not a core AI model release or research paper.
- 200 OK
- 402 Payment Required
- 403 Forbidden
- Cloudflare
- HTTP
- Pay Per Crawl
- RFC 9110: HTTP Semantics
- Stack Overflow
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →