Google Adds 41 AI Crawlers, Including GPTBot and ClaudeBot, to Its Maps robots.txt Opt-Out
TLDR
Google's robots.txt now lists 41 AI crawlers, including GPTBot and ClaudeBot, under the AI training opt-out Google backed in October, while its own Gemini control is not included.
What this Google Business Profile policy change means for your site
- Status
- Policy change
- Platform
- Google Business Profile
- Affects
- robots.txt AI training opt-out, Maps crawling
- Who
- Not specified in the source
- Effective
- Not specified in the source
- Action needed
- Not specified in the source
Optimixed Overview
Optimixed's original report on this story.
PPC Land reports that Google's robots.txt file now places 41 AI crawlers, including GPTBot and ClaudeBot, under the AI training opt-out Google backed in October. Google's own Gemini control sits outside that group, and the coverage raises the question of why the split exists.
How we got here: earlier Google Business Profile coverage
On October 21, Google clarified that robots.txt controls which bots can crawl a site, not what they do with the data they collect, and that non-standard rules are ignored under RFC 9309. Then, on October 31, Google and Bing said they would honor a shared robots.txt signal that blocks AI training. Now, Google's robots.txt lists 41 AI crawlers, including GPTBot and ClaudeBot, under that AI training opt-out, though its own Gemini control is not included.
Summary of earlier coverage
- Nov 17Shared IPs, in-page CAPTCHAs and add-to-cart calls still block Google-Agent at site firewalls
Sites that have allowed Google-Agent can still block it through shared IPs, in-page CAPTCHAs and add-to-cart calls at their firewalls, and robots.txt generally does not apply.
- Oct 31Google and Bing will honor a shared robots.txt signal to block AI training
Google and Bing said they will honor a shared robots.txt signal that blocks AI training, and a study put AI Overviews click-through at 0.9%.
- Oct 21Google clarifies that robots.txt controls crawler access, not how AI uses indexed data
Google clarified that robots.txt controls which bots can crawl a site, not what those bots do with the data they collect, and that non-standard rules are silently ignored under RFC 9309.
- Sep 15Cloudflare's Disallow AI Training feature allows sites to block AI scrapers without excluding Googlebot
Cloudflare introduced a Disallow AI Training option that blocks AI crawlers from training data while preserving access for Googlebot and other search engines.