A user reported that Anthropic's Claude AI model is unable to access their WordPress website, encountering errors such as 429 (Too Many Requests), ROBOTS_DISALLOWED, and DNS resolution failures. Despite the site's robots.txt file allowing access, Claude's web fetch tool initially reported the host as not being on an allowlist, and later failed to resolve the host. The user noted that other AI models like ChatGPT and DeepSeek can access the site without issue, suggesting the problem might stem from Claude's specific web fetching mechanism, the hosting provider (DreamHost), or a bot rule. AI
IMPACT Indicates potential limitations in AI models' ability to reliably access and process information from specific web platforms.
RANK_REASON User-reported issue with an AI model's web-fetching capabilities.
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
