Google to Support Shared AI Training Control in robots.txt Across AI Providers
TLDR
Google plans to support a shared AI training control in robots.txt that lets websites state AI training preferences for multiple AI providers in a single step, based on IETF AIPREF working group work.
Optimixed Overview
Optimixed's original report on this story.
Google Search Central Blog announces plans to support a shared AI training control in robots.txt, which would let website owners express their AI training preferences across multiple AI providers in one step. The control is based on work from the AIPREF working group at the Internet Engineering Task Force and builds on Google-Extended, the AI training control Google has offered since 2023.
How we got here
- Oct 16Meta's Updated Terms Allow Crawling Business Websites for AI Training Starting Oct. 30
Meta's updated terms, effective Oct. 30, grant the company rights to crawl websites tied to Facebook or Instagram business accounts for AI training purposes.
- Sep 15Cloudflare's Disallow AI Training feature allows sites to block AI scrapers without excluding Googlebot
Cloudflare introduced a Disallow AI Training option that blocks AI crawlers from training data while preserving access for Googlebot and other search engines.
- Aug 28WebMCP Enables AI Agents to Interact With Websites Through Structured Actions
OpenAI, Shopify, and Cloudflare are deploying WebMCP to enable AI agents to interact directly with website actions without relying on visual automation.
- Aug 28AI Is Reshaping Website Security: What SEO Teams Need to Know Now
Over 100 organizations including OpenAI, Google, and Microsoft are warning website owners to prepare for escalating AI-driven cyberattacks.