Google and Bing will honor a shared robots.txt signal to block AI training
TLDR
Google and Bing said they will honor a shared robots.txt signal that blocks AI training, and a study put AI Overviews click-through at 0.9%.
What this means for your site
- Status
- Policy change
- Platform
- Google Search, Bing
- Affects
- robots.txt opt-out signal for AI training
- Who
- site owners who control crawling via robots.txt
- Effective
- Not specified in the source
- Action needed
- Not specified in the source
Optimixed Overview
Optimixed's original report on this story.
Google and Bing said they will honor a shared robots.txt signal to block AI training. Separately, a study put AI Overviews click-through at 0.9%, and Bing has tested ads, as first reported by PPC Land.
How we got here
- Oct 30Google Is Adopting New AI Training Controls Under IETF Consideration
Google is adopting new AI training controls, currently under consideration by the IETF, to simplify and streamline how publishers and site owners manage AI training.
- Oct 30Study Puts Google AI Overviews and AI Mode Click-Through Rate at 0.9%
A new small study puts the click-through rate from Google's AI Overviews and AI Mode at about 0.9%, and finds that opting out of generative AI results also reduces clicks in the main web results.
- Oct 30Google to Support Shared AI Training Control in robots.txt Across AI Providers
Google plans to support a shared AI training control in robots.txt that lets websites state AI training preferences for multiple AI providers in a single step, based on IETF AIPREF working group work.
- Sep 15Cloudflare's Disallow AI Training feature allows sites to block AI scrapers without excluding Googlebot
Cloudflare introduced a Disallow AI Training option that blocks AI crawlers from training data while preserving access for Googlebot and other search engines.