A study of 100 well-known websites revealed varying levels of AI crawler readiness. While 38% of sites served an `llms.txt` file, this was significantly more common among developer tools than news or consumer sites. Furthermore, 25% of sites blocked at least one of the major AI crawlers (GPTBot, ClaudeBot, PerplexityBot) via their `robots.txt` files, with news sites showing a particularly high rate of blocking for ClaudeBot. The research also noted differences in how sites configure access for search crawlers versus those used for foundation model training. AI
IMPACT Highlights varying levels of AI data access control across the web, potentially impacting model training and search result inclusion.
RANK_REASON Research study analyzing website configurations for AI crawlers. [lever_c_demoted from research: ic=1 ai=1.0]
- Anthropic
- ChatGPT
- ClaudeBot
- GPTBot
- Jeremy Howard
- llms.txt
- OAI-SearchBot
- OpenAI
- PerplexityBot
- robots.txt
- SEOBolt
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →