Major AI companies including Google, OpenAI, Microsoft, and Anthropic have announced plans to honor a shared robots.txt signal for AI training opt-outs. This initiative, discussed at the Internet Engineering Task Force, aims to provide publishers with a single instruction for AI training preferences across participating providers. While Anthropic has begun implementation, OpenAI expects it in the coming months, and Google and Microsoft are targeting early 2027. The proposal uses a new user-agent token, __usage_ai-training, to allow websites to specify which content can be used for training generative AI models, though it does not affect general crawling or prevent content from appearing in search results. AI
IMPACT Establishes a unified opt-out mechanism for AI training, potentially impacting data availability for model development.
RANK_REASON Major AI companies announce a coordinated policy change regarding AI training data. [lever_c_demoted from significant: ic=1 ai=0.4]
Read on dev.to — Anthropic tag →
- Anthropic
- Gemini
- Gemini Apps
- Internet Engineering Task Force
- Kevin Kelley
- Microsoft
- OpenAI
- robots.txt
- Vertex AI
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →