A website operator has detailed an ongoing battle against AI scrapers, reporting that over 99% of their website traffic originates from bots. The operator found that for every human visitor, AI scrapers like Anthropic's Claude-SearchBot made approximately 214 page requests. This extensive scraping, particularly from sources like Claude-SearchBot and Amzn-SearchBot, occurred without generating any discernible human visitors, prompting the operator to implement strict firewall rules and challenges to block non-contributing AI traffic. AI
IMPACT Aggressive AI scraping strains website resources and may necessitate new payment models for data access.
RANK_REASON Operator's detailed account of dealing with AI bot traffic, not a direct release from an AI lab.
Read on Mastodon — mastodon.social →
- Amzn-SearchBot
- Anthropic
- Bingbot
- Chrome DevTools MCP
- Claude-SearchBot
- Cloudflare
- Googlebot
- Mastodon
- Nick Gray
- PatronView
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
