PulseAugur
EN
LIVE 09:57:53

AI Bots Ignore Robots.txt, Attempt Database Scans

By PulseAugur Editorial · [1 sources] ·

Several AI-driven web crawlers, including those from Anthropic's Claude and OpenAI's GPT bot, have been observed ignoring robots.txt directives and attempting to scan databases. These bots, along with others from Baidu, Amazon, Meta, and Yandex, were blocked by the server administrator. The administrator expressed frustration, stating that these large corporations are attempting to steal resources and that a simultaneous surge of these bots could render servers unusable, citing a recent incident with their PieFed server. AI

IMPACT AI crawlers are aggressively scraping data, potentially impacting server resources and data privacy for smaller platforms.

RANK_REASON This is a user's complaint about AI bots, not a direct release or announcement from a frontier lab.

Read on Mastodon — fosstodon.org →

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →

AI Bots Ignore Robots.txt, Attempt Database Scans

How we ranked this

Signal score
0 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Commentary
This is a user's complaint about AI bots, not a direct release or announcement from a frontier lab.
Source corroboration
Single-source cluster
Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics
other
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
101 days old
Aged out of breaking-news scoring windows; ranking reflects the durable signal from the full source set.

Full methodology in our editorial standards.

COVERAGE [1]

  1. Mastodon — fosstodon.org TIER_1 English(EN) · [email protected] ·

    In the past 24 hours, these spiders have ignored my robots.txt file and tried scanning the database anyway. They were all blocked. Claude (thousands more attemp

    In the past 24 hours, these spiders have ignored my robots.txt file and tried scanning the database anyway. They were all blocked. Claude (thousands more attempts than the others) Baiduspider Amazonbot Bytespider gptbot Meta-ExternalAgent YandexBot ChatGPT ByteSpider CommonCrawl …

RELATED ENTITIES

RELATED TOPICS