The HTTP Code Your AI Agent Doesn't Handle Yet: 402
Cloudflare has begun implementing the HTTP 402 Payment Required status code for its Pay-Per-Crawl service, a move that significantly alters how web crawlers interact with websites. Previously, crawlers faced 200 OK for access or 403 Forbidden for restrictions. The new 402 code acts as a quote for a fee, requiring crawlers to re-request with an exact price to gain access, effectively turning access control into a transactional process. A naive implementation of this new code can lead to significantly overspending, as demonstrated by a test where a simple agent spent 10x its budget, while a more sophisticated handler with a price cap and fallback options managed costs effectively and retrieved fewer pages. AI
IMPACT Web crawlers and AI agents will need new logic to handle transactional access, impacting data acquisition costs and strategies.