In the past 24 hours, these spiders have ignored my robots.txt file and tried scanning the database anyway. They were all blocked. Claude (thousands more attemp
Several AI-driven web crawlers, including those from Anthropic's Claude and OpenAI's GPT bot, have been observed ignoring robots.txt directives and attempting to scan databases. These bots, along with others from Baidu, Amazon, Meta, and Yandex, were blocked by the server administrator. The administrator expressed frustration, stating that these large corporations are attempting to steal resources and that a simultaneous surge of these bots could render servers unusable, citing a recent incident with their PieFed server. AI
IMPACT AI crawlers are aggressively scraping data, potentially impacting server resources and data privacy for smaller platforms.