User deploys Anubis to block AI scrapers ignoring robots.txt

By PulseAugur Editorial · [1 sources] · 2026-06-01 07:52

A Mastodon user expressed frustration with AI scrapers that disregard robots.txt, leading them to implement Anubis to protect their websites. The user believes AI companies have broken a "moral contract" by not respecting these directives, justifying defensive measures. AI

RANK_REASON User-generated content on a social media platform about a tool, not a significant industry event.

Read on Mastodon — mastodon.social →

other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →

COVERAGE [1]

Mastodon — mastodon.social TIER_1 English(EN) · rotnroll666 · 2026-06-01 07:52

Obviously, there's an innuendo towards # jqwik in the above post. Think about it: I need to protect all my sites that don't just deliver plain HTTP files with #

Obviously, there's an innuendo towards # jqwik in the above post. Think about it: I need to protect all my sites that don't just deliver plain HTTP files with # Anubis these days. Non of the fucking # AI scrapers obeys robots.txt (the equivalent of "pretty please"). Anubis is rea…

COVERAGE [1]

Obviously, there's an innuendo towards # jqwik in the above post. Think about it: I need to protect all my sites that don't just deliver plain HTTP files with #

RELATED ENTITIES

RELATED TOPICS