robots.txt
PulseAugur coverage of robots.txt — every cluster mentioning robots.txt across labs, papers, and developer communities, ranked by signal.
- 2026-11-01 regulatory Google, Microsoft, and OpenAI are collaborating to establish a standardized method for controlling AI data training via robots.txt, with a planned implementation by 2027. source
15 day(s) with sentiment data
New bot directive file standard emerges beyond llms.txt
The success of Anna's Archive's llms.txt suggests a growing need for more nuanced bot directives than robots.txt offers. It's plausible that other organizations will adopt or create similar convention-based files to guide AI crawlers for specific purposes, potentially leading to a new de facto standard for AI-specific web access control.
Websites increasingly block AI crawlers via IP ranges, not just robots.txt
Evidence shows users are actively exploring and recommending blocking Google's AI search scans via IP ranges, rather than solely relying on robots.txt. This indicates a shift in strategy as websites become wary of AI crawlers' impact and the perceived inadequacy of robots.txt for controlling AI-specific access.
Google to deprecate robots.txt for AI crawlers due to complexity
Given the documented issues with Google's crawler documentation and the increasing complexity of AI content access needs, it's plausible Google may eventually move away from relying solely on robots.txt for its AI crawlers. They might introduce a more sophisticated, AI-specific directive system or API to manage access, especially as they shift to an AI-first search model.
-
Websites show mixed AI crawler readiness, with many blocking bots
A study of 100 well-known websites revealed varying levels of AI crawler readiness. While 38% of sites served an `llms.txt` file, this was significantly more common among developer tools than news or consumer sites. Fur…
-
Major AI firms to respect robots.txt signals; Australia mandates safety standards · 2 sources tracked
Three major AI companies, including OpenAI, Anthropic, and Google, have pledged to honor new preference signals within robots.txt files. These signals will allow website owners to more clearly indicate which content is …
-
New ScrapeBench benchmark reveals low legal compliance in AI agents
A new benchmark called ScrapeBench has been developed to evaluate the legal compliance of AI agents when scraping websites. Researchers found that most AI agents, both closed- and open-weight, have low compliance rates …
-
Developer details method to block AI bots by over 80% using honeypot
A developer has detailed a method to block unwanted AI bot traffic from websites, reducing it by over 80%. The technique involves creating a hidden "honeypot" directory on a website and instructing bots via robots.txt n…
-
Meta delays AI training terms update amid global backlash
Meta has indefinitely postponed a controversial update to its commercial terms that would have allowed the company to train its AI models on content from external websites linked on Facebook. The company faced significa…
-
Mastodon updates robots.txt to control AI data usage
Mastodon has updated its robots.txt file to provide guidelines for AI crawlers. The update allows AI search engines to access content while explicitly disallowing AI training on the platform's data. This move aims to gi…
-
robots.txt proposal aims to block all AI training bots
A proposal suggests using a single entry in the robots.txt file to block all AI training bots from accessing websites. This method aims to provide a unified way for website owners to prevent unauthorized data scraping f…
-
Google, OpenAI, Microsoft, Anthropic to honor shared AI training opt-out · 1 source tracked
Major AI companies including Google, OpenAI, Microsoft, and Anthropic have announced plans to honor a shared robots.txt signal for AI training opt-outs. This initiative, discussed at the Internet Engineering Task Force,…
-
Google, Microsoft, OpenAI to Standardize AI Training Controls via robots.txt by 2027
Major technology companies including Google, Microsoft, and OpenAI are collaborating to establish a standardized method for controlling AI data training. They plan to implement a common specification via robots.txt by 2…
-
OpenAI, Google, Bing back robots.txt signal for AI training preferences · 7 sources tracked
OpenAI, Google, and Microsoft Bing are adopting a new robots.txt signal, __usage_ai-training, to allow website owners to express preferences about using their content for AI model training. This signal is intended as a …
-
Google, OpenAI, Anthropic, Microsoft Pledge New AI Scraping Controls · 3 sources tracked
Google, OpenAI, Anthropic, and Microsoft have committed to supporting a new standard within robots.txt files. This standard aims to prevent AI companies from scraping web content for training their models. The initiativ…
-
Google, OpenAI, Anthropic adopt new AI scraping controls · 2 sources tracked
Google, OpenAI, and Anthropic are adopting a new robots.txt directive to control AI training data scraping. Proposed by Anthropic's Kevin Kelley, the `__usage_ai-training` user agent will signal to AI companies which we…
-
Google, Bing to Honor Robots.txt for AI Training Opt-Out
Google and Microsoft Bing have announced they will respect the robots.txt protocol to prevent their AI models from being trained on website content. This move comes after a study indicated that AI Overviews are signific…
-
OpenAI to honor robots.txt for AI training data preferences
OpenAI is introducing a new method to respect website owners' preferences regarding AI training data. By honoring a shared robots.txt signal, OpenAI aims to allow website operators to communicate their desired data usag…
-
New llms.txt format aims to guide AI crawlers to key website pages
The llms.txt format is a community-driven proposal for websites to provide AI tools with a curated list of important pages. This Markdown file, typically located at the root of a website, includes a site title, a summar…
-
AI crawlers face varied robots.txt rules; Distill prioritizes explicit opt-outs
Hybrid Forge Lab, the creators of the Distill URL-to-Markdown API, have analyzed robots.txt files from 37 major websites to understand how AI crawlers are treated. Their findings indicate that while specific AI crawler …
-
GEO and SEO are distinct metrics; agent recipes improve AI reliability
A new approach to Generative Engine Optimization (GEO) distinguishes it from traditional Search Engine Optimization (SEO), arguing that GEO scores measure content's quotability and machine-parseability, while SEO focuse…
-
EU regulators warn AI firms on data scraping; draft bill proposes model halts
The European Data Protection Board (EDPB) has advised AI developers to avoid scraping websites that block data collection via robots.txt, outlining six minimization steps. Separately, a draft bill, potentially linked to…
-
Meta revises AI TOS after backlash, but still won't respect robots.txt
Meta has revised its commercial Terms of Service to remove the word "train" following significant backlash from news organizations and legal experts. Despite this change, Meta has not agreed to respect robots.txt files,…
-
OpenAI shutters Sora 2, Emirates NBD integrates models, Anthropic crawler details emerge
OpenAI has reportedly shut down its Sora 2 model and the associated Videos API, with code calls now failing. Separately, Emirates NBD is integrating OpenAI models with local GPUs in the UAE, becoming the first bank in t…