Gary Illyes Says Spoofing Googlebot Takes About 30 Seconds, Exposing Gaps in User-Agent Checks
TLDR
Google's Gary Illyes says impersonating Googlebot takes about 30 seconds, and Google signs only some Google-Agent requests, leaving site owners who rely on user-agent headers unable to spot impostors.
Optimixed Overview
Optimixed's original report on this story.
Google's Gary Illyes says faking Googlebot takes about 30 seconds, which means site owners who trust user-agent headers cannot reliably identify impostor crawlers. Google signs only some Google-Agent requests, and the article raises whether a bot reputation system could close that gap, as first reported by PPC Land.
How we got here: earlier coverage
- Oct 30Google's Gary Illyes Says AI Is Making Bot Impersonation a Bigger Problem
Google's Gary Illyes said in a podcast interview that AI has made bot impersonation, where rogue bots pose as Googlebot, a bigger problem.
- Oct 22Google CEO Says AI Agents Will Access Websites Through Pages and APIs
Google's CEO indicated AI agents will access websites both through traditional page browsing and via APIs, with protocols offering better scalability but longer development cycles.
- Oct 21Google clarifies that robots.txt controls crawler access, not how AI uses indexed data
Google clarified that robots.txt controls which bots can crawl a site, not what those bots do with the data they collect, and that non-standard rules are silently ignored under RFC 9309.
- Oct 17Cloudflare will penalize verified bots that ignore robots.txt and rate-limit signals with lower trust scores
Cloudflare is planning to assign lower trust scores to verified bots that disregard robots.txt directives or ignore HTTP 429 rate-limiting responses.