PulseAugur
EN
LIVE 10:24:20
ENTITY robots.txt

robots.txt

PulseAugur coverage of robots.txt — every cluster mentioning robots.txt across labs, papers, and developer communities, ranked by signal.

Show in brief
Total · 30d
26
62 over 90d
Releases · 30d
0
0 over 90d
Papers · 30d
1
2 over 90d
TIER MIX · 90D
TOPICS
RELATIONSHIPS
TIMELINE
  1. 2026-11-01 regulatory Google, Microsoft, and OpenAI are collaborating to establish a standardized method for controlling AI data training via robots.txt, with a planned implementation by 2027. source
SENTIMENT · 30D

15 day(s) with sentiment data

LAB BRAIN
hypothesis resolved confirmed conf 0.60

New bot directive file standard emerges beyond llms.txt

The success of Anna's Archive's llms.txt suggests a growing need for more nuanced bot directives than robots.txt offers. It's plausible that other organizations will adopt or create similar convention-based files to guide AI crawlers for specific purposes, potentially leading to a new de facto standard for AI-specific web access control.

observation resolved confirmed conf 0.70

Websites increasingly block AI crawlers via IP ranges, not just robots.txt

Evidence shows users are actively exploring and recommending blocking Google's AI search scans via IP ranges, rather than solely relying on robots.txt. This indicates a shift in strategy as websites become wary of AI crawlers' impact and the perceived inadequacy of robots.txt for controlling AI-specific access.

hypothesis resolved contradicted conf 0.55

Google to deprecate robots.txt for AI crawlers due to complexity

Given the documented issues with Google's crawler documentation and the increasing complexity of AI content access needs, it's plausible Google may eventually move away from relying solely on robots.txt for its AI crawlers. They might introduce a more sophisticated, AI-specific directive system or API to manage access, especially as they shift to an AI-first search model.

All hypotheses →

RECENT · PAGE 1/4 · 70 TOTAL
  1. TOOL · CL_366758 ·

    Websites show mixed AI crawler readiness, with many blocking bots

    A study of 100 well-known websites revealed varying levels of AI crawler readiness. While 38% of sites served an `llms.txt` file, this was significantly more common among developer tools than news or consumer sites. Fur…

  2. RESEARCH · CL_352775 ·

    Major AI firms to respect robots.txt signals; Australia mandates safety standards · 2 sources tracked

    Three major AI companies, including OpenAI, Anthropic, and Google, have pledged to honor new preference signals within robots.txt files. These signals will allow website owners to more clearly indicate which content is …

  3. RESEARCH · CL_352156 ·

    New ScrapeBench benchmark reveals low legal compliance in AI agents

    A new benchmark called ScrapeBench has been developed to evaluate the legal compliance of AI agents when scraping websites. Researchers found that most AI agents, both closed- and open-weight, have low compliance rates …

  4. TOOL · CL_349066 ·

    Developer details method to block AI bots by over 80% using honeypot

    A developer has detailed a method to block unwanted AI bot traffic from websites, reducing it by over 80%. The technique involves creating a hidden "honeypot" directory on a website and instructing bots via robots.txt n…

  5. RESEARCH · CL_345485 ·

    Meta delays AI training terms update amid global backlash

    Meta has indefinitely postponed a controversial update to its commercial terms that would have allowed the company to train its AI models on content from external websites linked on Facebook. The company faced significa…

  6. TOOL · CL_340443 ·

    Mastodon updates robots.txt to control AI data usage

    Mastodon has updated its robots.txt file to provide guidelines for AI crawlers. The update allows AI search engines to access content while explicitly disallowing AI training on the platform's data. This move aims to gi…

  7. COMMENTARY · CL_340070 ·

    robots.txt proposal aims to block all AI training bots

    A proposal suggests using a single entry in the robots.txt file to block all AI training bots from accessing websites. This method aims to provide a unified way for website owners to prevent unauthorized data scraping f…

  8. RESEARCH · CL_336025 ·

    Google, OpenAI, Microsoft, Anthropic to honor shared AI training opt-out · 1 source tracked

    Major AI companies including Google, OpenAI, Microsoft, and Anthropic have announced plans to honor a shared robots.txt signal for AI training opt-outs. This initiative, discussed at the Internet Engineering Task Force,…

  9. RESEARCH · CL_334582 ·

    Google, Microsoft, OpenAI to Standardize AI Training Controls via robots.txt by 2027

    Major technology companies including Google, Microsoft, and OpenAI are collaborating to establish a standardized method for controlling AI data training. They plan to implement a common specification via robots.txt by 2…

  10. SIGNIFICANT · CL_334395 ·

    OpenAI, Google, Bing back robots.txt signal for AI training preferences · 7 sources tracked

    OpenAI, Google, and Microsoft Bing are adopting a new robots.txt signal, __usage_ai-training, to allow website owners to express preferences about using their content for AI model training. This signal is intended as a …

  11. SIGNIFICANT · CL_333319 ·

    Google, OpenAI, Anthropic, Microsoft Pledge New AI Scraping Controls · 3 sources tracked

    Google, OpenAI, Anthropic, and Microsoft have committed to supporting a new standard within robots.txt files. This standard aims to prevent AI companies from scraping web content for training their models. The initiativ…

  12. SIGNIFICANT · CL_333309 ·

    Google, OpenAI, Anthropic adopt new AI scraping controls · 2 sources tracked

    Google, OpenAI, and Anthropic are adopting a new robots.txt directive to control AI training data scraping. Proposed by Anthropic's Kevin Kelley, the `__usage_ai-training` user agent will signal to AI companies which we…

  13. RESEARCH · CL_332616 ·

    Google, Bing to Honor Robots.txt for AI Training Opt-Out

    Google and Microsoft Bing have announced they will respect the robots.txt protocol to prevent their AI models from being trained on website content. This move comes after a study indicated that AI Overviews are signific…

  14. TOOL · CL_331860 ·

    OpenAI to honor robots.txt for AI training data preferences

    OpenAI is introducing a new method to respect website owners' preferences regarding AI training data. By honoring a shared robots.txt signal, OpenAI aims to allow website operators to communicate their desired data usag…

  15. TOOL · CL_329840 ·

    New llms.txt format aims to guide AI crawlers to key website pages

    The llms.txt format is a community-driven proposal for websites to provide AI tools with a curated list of important pages. This Markdown file, typically located at the root of a website, includes a site title, a summar…

  16. TOOL · CL_328338 ·

    AI crawlers face varied robots.txt rules; Distill prioritizes explicit opt-outs

    Hybrid Forge Lab, the creators of the Distill URL-to-Markdown API, have analyzed robots.txt files from 37 major websites to understand how AI crawlers are treated. Their findings indicate that while specific AI crawler …

  17. COMMENTARY · CL_323620 ·

    GEO and SEO are distinct metrics; agent recipes improve AI reliability

    A new approach to Generative Engine Optimization (GEO) distinguishes it from traditional Search Engine Optimization (SEO), arguing that GEO scores measure content's quotability and machine-parseability, while SEO focuse…

  18. RESEARCH · CL_313177 ·

    EU regulators warn AI firms on data scraping; draft bill proposes model halts

    The European Data Protection Board (EDPB) has advised AI developers to avoid scraping websites that block data collection via robots.txt, outlining six minimization steps. Separately, a draft bill, potentially linked to…

  19. SIGNIFICANT · CL_311640 ·

    Meta revises AI TOS after backlash, but still won't respect robots.txt

    Meta has revised its commercial Terms of Service to remove the word "train" following significant backlash from news organizations and legal experts. Despite this change, Meta has not agreed to respect robots.txt files,…

  20. TOOL · CL_310290 ·

    OpenAI shutters Sora 2, Emirates NBD integrates models, Anthropic crawler details emerge

    OpenAI has reportedly shut down its Sora 2 model and the associated Videos API, with code calls now failing. Separately, Emirates NBD is integrating OpenAI models with local GPUs in the UAE, becoming the first bank in t…