Optimixed Search News

Google Adds 41 AI Crawlers, Including GPTBot and ClaudeBot, to Its Maps robots.txt Opt-Out

By Eric Mandell
65
News

TLDR

Google's robots.txt now lists 41 AI crawlers, including GPTBot and ClaudeBot, under the AI training opt-out Google backed in October, while its own Gemini control is not included.

Context:3 closely related earlier stories since October

What this Google Business Profile policy change means for your site

Status
Policy change
Platform
Google Business Profile
Affects
robots.txt AI training opt-out, Maps crawling
Who
Not specified in the source
Effective
Not specified in the source
Action needed
Not specified in the source

Optimixed Overview

Optimixed's original report on this story.

PPC Land reports that Google's robots.txt file now places 41 AI crawlers, including GPTBot and ClaudeBot, under the AI training opt-out Google backed in October. Google's own Gemini control sits outside that group, and the coverage raises the question of why the split exists.

How we got here: earlier Google Business Profile coverage

On October 21, Google clarified that robots.txt controls which bots can crawl a site, not what they do with the data they collect, and that non-standard rules are ignored under RFC 9309. Then, on October 31, Google and Bing said they would honor a shared robots.txt signal that blocks AI training. Now, Google's robots.txt lists 41 AI crawlers, including GPTBot and ClaudeBot, under that AI training opt-out, though its own Gemini control is not included.

Summary of earlier coverage

  1. Nov 17
    Shared IPs, in-page CAPTCHAs and add-to-cart calls still block Google-Agent at site firewalls

    Sites that have allowed Google-Agent can still block it through shared IPs, in-page CAPTCHAs and add-to-cart calls at their firewalls, and robots.txt generally does not apply.

  2. Oct 31
    Google and Bing will honor a shared robots.txt signal to block AI training

    Google and Bing said they will honor a shared robots.txt signal that blocks AI training, and a study put AI Overviews click-through at 0.9%.

  3. Oct 21
    Google clarifies that robots.txt controls crawler access, not how AI uses indexed data

    Google clarified that robots.txt controls which bots can crawl a site, not what those bots do with the data they collect, and that non-standard rules are silently ignored under RFC 9309.

  4. Sep 15
    Cloudflare's Disallow AI Training feature allows sites to block AI scrapers without excluding Googlebot

    Cloudflare introduced a Disallow AI Training option that blocks AI crawlers from training data while preserving access for Googlebot and other search engines.

Was this useful?
Share

More AI & Machine Learning in SEO stories

68
News

After one site used Google's Search Console opt-out from AI Overviews and AI Mode, hundreds of thousands of daily impressions disappeared while clicks barely moved, according to Brodie Clark.

AI & Machine Learning in SEOSEO News & Algorithm UpdatesAnalytics & Measurement
72
News

Turkey has issued an October 8 interim order requiring Google to let news publishers opt out of AI Overviews while staying in search, with non-discrimination and value-based terms also granted to publishers.

AI & Machine Learning in SEOSEO News & Algorithm Updates
45
News

Google AI Overviews and ChatGPT now provide merchant coupon and promo codes directly in their answers, so shoppers can copy codes without visiting coupon sites.

AI & Machine Learning in SEOEcommerce MarketingSEO News & Algorithm Updates
50
News

OpenAI revised the terms of its new ChatGPT Business Hub days after launch, following Digiday reporting that the original language could limit businesses' ability to bring legal claims.

AI & Machine Learning in SEOContent & StrategyAnalytics & Measurement