Shared IPs, in-page CAPTCHAs and add-to-cart calls still block Google-Agent at site firewalls
TLDR
Sites that have allowed Google-Agent can still block it through shared IPs, in-page CAPTCHAs and add-to-cart calls at their firewalls, and robots.txt generally does not apply.
Optimixed Overview
Optimixed's original report on this story.
Sites can keep blocking Google-Agent even after permitting it, because shared IPs, in-page CAPTCHAs and add-to-cart calls trip the agent up at site firewalls. PPC Land notes that robots.txt generally does not apply to the agent, which leaves open what control sites retain over its access.
How we got here: earlier coverage
On October 21, Google clarified that robots.txt controls which bots can crawl a site, not what those bots do with the data they collect. On November 5, bot protection blocked Blend's AI shopping agent at 29 of 200 retailers, with large retailers refusing it far more often than emerging ones. Now, sites that have allowed Google-Agent can still block it through shared IPs, in-page CAPTCHAs and add-to-cart calls at their firewalls, and robots.txt generally does not apply.
Summary of earlier coverage
- Nov 5Large Retailers Block Blend's AI Shopping Agent More Often Than Smaller Ones
Bot protection blocked Blend's AI shopping agent at 29 of 200 retailers, with large retailers refusing it far more often than emerging ones.
- Oct 22Google CEO Says AI Agents Will Access Websites Through Pages and APIs
Google's CEO indicated AI agents will access websites both through traditional page browsing and via APIs, with protocols offering better scalability but longer development cycles.
- Oct 21Google clarifies that robots.txt controls crawler access, not how AI uses indexed data
Google clarified that robots.txt controls which bots can crawl a site, not what those bots do with the data they collect, and that non-standard rules are silently ignored under RFC 9309.
- Sep 15Cloudflare's Disallow AI Training feature allows sites to block AI scrapers without excluding Googlebot
Cloudflare introduced a Disallow AI Training option that blocks AI crawlers from training data while preserving access for Googlebot and other search engines.