Optimixed Search News

Shared IPs, in-page CAPTCHAs and add-to-cart calls still block Google-Agent at site firewalls

By Eric Mandell
62
News

TLDR

Sites that have allowed Google-Agent can still block it through shared IPs, in-page CAPTCHAs and add-to-cart calls at their firewalls, and robots.txt generally does not apply.

Context:5 closely related earlier stories since September

Optimixed Overview

Optimixed's original report on this story.

Sites can keep blocking Google-Agent even after permitting it, because shared IPs, in-page CAPTCHAs and add-to-cart calls trip the agent up at site firewalls. PPC Land notes that robots.txt generally does not apply to the agent, which leaves open what control sites retain over its access.

How we got here: earlier coverage

On October 21, Google clarified that robots.txt controls which bots can crawl a site, not what those bots do with the data they collect. On November 5, bot protection blocked Blend's AI shopping agent at 29 of 200 retailers, with large retailers refusing it far more often than emerging ones. Now, sites that have allowed Google-Agent can still block it through shared IPs, in-page CAPTCHAs and add-to-cart calls at their firewalls, and robots.txt generally does not apply.

Summary of earlier coverage

  1. Nov 5
    Large Retailers Block Blend's AI Shopping Agent More Often Than Smaller Ones

    Bot protection blocked Blend's AI shopping agent at 29 of 200 retailers, with large retailers refusing it far more often than emerging ones.

  2. Oct 22
    Google CEO Says AI Agents Will Access Websites Through Pages and APIs

    Google's CEO indicated AI agents will access websites both through traditional page browsing and via APIs, with protocols offering better scalability but longer development cycles.

  3. Oct 21
    Google clarifies that robots.txt controls crawler access, not how AI uses indexed data

    Google clarified that robots.txt controls which bots can crawl a site, not what those bots do with the data they collect, and that non-standard rules are silently ignored under RFC 9309.

  4. Sep 15
    Cloudflare's Disallow AI Training feature allows sites to block AI scrapers without excluding Googlebot

    Cloudflare introduced a Disallow AI Training option that blocks AI crawlers from training data while preserving access for Googlebot and other search engines.

Was this useful?
Share

More AI & Machine Learning in SEO stories

45
News

Google has released a new Search help document explaining how website owners can allow Google-Agent to access their sites.

Technical SEOSEO News & Algorithm Updates
52
News

Google's John Mueller reiterated that the Crawled/Discovered - Currently Not Indexed status in Search Console's page indexing report can indicate Google doesn't consider a site or its pages high-quality enough to index.

Technical SEOSEO News & Algorithm UpdatesAnalytics & Measurement
65
News

Google has clarified how server-side redirects influence ranking signals and canonicalization.

Technical SEOSEO News & Algorithm Updates
62
News

Google's Gary Illyes said Google Search can temporarily keep important URLs that return a 4xx status code in the index rather than removing them immediately.

Technical SEOSEO News & Algorithm Updates