An experiment was conducted to test Perplexity-Bot's adherence to webmaster directives. The bot was granted firewall access, and despite having access to robots.txt, it did not respect the crawl-delay directive. The experiment concluded with the observation that the bot is not used for generative AI training but rather as an indexer. AI
IMPACT This incident highlights potential issues with web crawler compliance, impacting how AI indexing services interact with website resources.
RANK_REASON The item describes a specific bot's behavior regarding web crawling directives, which falls under tooling or infrastructure rather than a core AI release or significant industry event.
Read on Mastodon — fosstodon.org →
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
