Google, OpenAI, and Anthropic are adopting a new robots.txt directive to control AI training data scraping. Proposed by Anthropic's Kevin Kelley, the `__usage_ai-training` user agent will signal to AI companies which websites do not want their content used for training generative models. While this prevents future scraping, it does not affect data already collected. Microsoft and Google plan to implement this in early 2027, while OpenAI will adopt it in the coming months. AI
IMPACT Establishes a common standard for AI companies to respect website owners' data usage preferences, potentially impacting future training datasets.
RANK_REASON Major tech companies adopting a new industry standard for data scraping.
AI-generated summary · Google Gemini · from 2 sources. How we write summaries →
