Optimixed Search News

Google and Bing will honor a shared robots.txt signal to block AI training

By Eric Mandell
70
News

TLDR

Google and Bing said they will honor a shared robots.txt signal that blocks AI training, and a study put AI Overviews click-through at 0.9%.

Context:The 12th Google Search policy change we've covered in the last 30 days

What this Google Search policy change means for your site

Status
Policy change
Platform
Google Search, Bing
Affects
robots.txt opt-out signal for AI training
Who
site owners who control crawling via robots.txt
Effective
Not specified in the source
Action needed
Not specified in the source

Optimixed Overview

Optimixed's original report on this story.

Google and Bing said they will honor a shared robots.txt signal to block AI training. Separately, a study put AI Overviews click-through at 0.9%, and Bing has tested ads, as first reported by PPC Land.

How we got here: earlier Google Search coverage

  1. Oct 30
    Google Is Adopting New AI Training Controls Under IETF Consideration

    Google is adopting new AI training controls, currently under consideration by the IETF, to simplify and streamline how publishers and site owners manage AI training.

  2. Oct 30
    Study Puts Google AI Overviews and AI Mode Click-Through Rate at 0.9%

    A new small study puts the click-through rate from Google's AI Overviews and AI Mode at about 0.9%, and finds that opting out of generative AI results also reduces clicks in the main web results.

  3. Oct 30
    Google to Support Shared AI Training Control in robots.txt Across AI Providers

    Google plans to support a shared AI training control in robots.txt that lets websites state AI training preferences for multiple AI providers in a single step, based on IETF AIPREF working group work.

  4. Sep 15
    Cloudflare's Disallow AI Training feature allows sites to block AI scrapers without excluding Googlebot

    Cloudflare introduced a Disallow AI Training option that blocks AI crawlers from training data while preserving access for Googlebot and other search engines.

Was this useful?
Share

More AI & Machine Learning in SEO stories

68
News

Every Google Search user now gets the Gemini 4 Hydrogen model, priced like Flash, after Pro and Ultra subscribers received the Argon version in AI Mode on October 13.

AI & Machine Learning in SEOSEO News & Algorithm Updates
72
News

Google says its Search has received a major upgrade powered by the Gemini 4 Hydrogen model, which it describes as faster with better understanding, query fan-out, linking, and visual responses.

AI & Machine Learning in SEOSEO News & Algorithm UpdatesTools & Platforms
50
News

Anthropic has banned fake websites built with Claude that are aimed at gaming AI-generated answers, while loosening some other usage restrictions.

AI & Machine Learning in SEOSEO News & Algorithm Updates
58
News

Andreessen Horowitz says AI agent traffic grew eightfold in two months, and that firms need to court these agents; October data from HUMAN customer sites shows Meta's Muse made 59% of agent requests and OpenAI 30%.

AI & Machine Learning in SEOCareer & Industry Trends