Google, Microsoft and OpenAI Plan to Honor a Shared robots.txt Line for Opting Out of AI Training
TLDR
A shared robots.txt line could let sites opt out of AI training with several providers at once, but the companies involved are handling it differently.
Optimixed Overview
Optimixed's original report on this story.
Search Engine Journal reports that a single shared robots.txt line could allow websites to opt out of AI training with several providers at the same time. The companies involved, however, are not yet handling the approach in the same way.
How we got here: earlier coverage
On October 30, Google said it planned to support a shared AI training control in robots.txt, based on IETF AIPREF working group work, that would let websites state AI training preferences for multiple AI providers in one step. On October 31, Google and Bing said they would honor a shared robots.txt signal that blocks AI training. Now, Google, Microsoft and OpenAI are planning a shared robots.txt AI training opt-out that could let sites opt out with several providers at once, though the companies are handling it differently.
Summary of earlier coverage
- Oct 31Sites Using NoArchive or NoCache Lose Microsoft's AI Opt-Out on March 1, 2027 Unless They Switch
Sites using NoArchive or NoCache will lose their Microsoft AI opt-out on March 1, 2027 unless they switch, and Google's token will also cover Gemini grounding.
- Oct 31Google and Bing will honor a shared robots.txt signal to block AI training
Google and Bing said they will honor a shared robots.txt signal that blocks AI training, and a study put AI Overviews click-through at 0.9%.
- Oct 30Google to Support Shared AI Training Control in robots.txt Across AI Providers
Google plans to support a shared AI training control in robots.txt that lets websites state AI training preferences for multiple AI providers in a single step, based on IETF AIPREF working group work.
- Sep 15Cloudflare's Disallow AI Training feature allows sites to block AI scrapers without excluding Googlebot
Cloudflare introduced a Disallow AI Training option that blocks AI crawlers from training data while preserving access for Googlebot and other search engines.