Google Explains What Robots.txt Does and Doesn't Control for AI Content Use
TLDR
Google clarified that robots.txt controls whether content gets crawled, not how it gets used after crawling, and recommends percent-encoding non-English robots.txt paths.
Optimixed Overview
Optimixed's original report on this story.
Google issued guidance distinguishing robots.txt's scope: the standard governs crawling access but does not dictate how content is subsequently used, including for AI training purposes. The company also recommended percent-encoding non-English characters in robots.txt file paths to ensure proper handling across systems.
How we got here: earlier coverage
On October 21, Google's Gary Illyes and Chery Prommawin released a video explaining which robots.txt directives Google and other search engines or AI crawlers actually support. The same day, Google clarified that robots.txt controls which bots can crawl a site, not what they do with the data they collect, and that non-standard rules are silently ignored under RFC 9309. Now, Google has clarified that robots.txt governs whether content gets crawled, not how it is used after crawling, and recommends percent-encoding non-English robots.txt paths.
Summary of earlier coverage
- Oct 21Google clarifies that robots.txt controls crawler access, not how AI uses indexed data
Google clarified that robots.txt controls which bots can crawl a site, not what those bots do with the data they collect, and that non-standard rules are silently ignored under RFC 9309.
- Oct 21Google Explains What's Actually Supported in robots.txt Files, Including AI Crawler Rules
Google's Gary Illyes and Chery Prommawin released a video explaining which robots.txt directives are actually supported by Google and other search engines or AI crawlers.
- Sep 18Cloudflare adds AI crawler management tool that lets sites set policy by category instead of individual bot
Cloudflare launched Bot Preference Sync, allowing sites to set AI crawler access policies by category rather than managing individual crawlers in robots.txt.
- Aug 15OpenAI Claims Robots.txt May Not Stop ChatGPT's Web-Crawling Bot
SEOFOMO News reports that OpenAI's documentation suggests ChatGPT's Fetch Bot may not be bound by robots.txt restrictions, despite evidence of it accessing blocked sites.