Optimixed Search News

Google Explains What Robots.txt Does and Doesn't Control for AI Content Use

By Eric Mandell
62
News

TLDR

Google clarified that robots.txt controls whether content gets crawled, not how it gets used after crawling, and recommends percent-encoding non-English robots.txt paths.

Context:4 closely related earlier stories since August

Optimixed Overview

Optimixed's original report on this story.

Google issued guidance distinguishing robots.txt's scope: the standard governs crawling access but does not dictate how content is subsequently used, including for AI training purposes. The company also recommended percent-encoding non-English characters in robots.txt file paths to ensure proper handling across systems.

How we got here: earlier coverage

On October 21, Google's Gary Illyes and Chery Prommawin released a video explaining which robots.txt directives Google and other search engines or AI crawlers actually support. The same day, Google clarified that robots.txt controls which bots can crawl a site, not what they do with the data they collect, and that non-standard rules are silently ignored under RFC 9309. Now, Google has clarified that robots.txt governs whether content gets crawled, not how it is used after crawling, and recommends percent-encoding non-English robots.txt paths.

Summary of earlier coverage

  1. Oct 21
    Google clarifies that robots.txt controls crawler access, not how AI uses indexed data

    Google clarified that robots.txt controls which bots can crawl a site, not what those bots do with the data they collect, and that non-standard rules are silently ignored under RFC 9309.

  2. Oct 21
    Google Explains What's Actually Supported in robots.txt Files, Including AI Crawler Rules

    Google's Gary Illyes and Chery Prommawin released a video explaining which robots.txt directives are actually supported by Google and other search engines or AI crawlers.

  3. Sep 18
    Cloudflare adds AI crawler management tool that lets sites set policy by category instead of individual bot

    Cloudflare launched Bot Preference Sync, allowing sites to set AI crawler access policies by category rather than managing individual crawlers in robots.txt.

  4. Aug 15
    OpenAI Claims Robots.txt May Not Stop ChatGPT's Web-Crawling Bot

    SEOFOMO News reports that OpenAI's documentation suggests ChatGPT's Fetch Bot may not be bound by robots.txt restrictions, despite evidence of it accessing blocked sites.

Was this useful?
Share

More AI & Machine Learning in SEO stories

62
News

An analysis of Search Console data from French sites links AI Overviews to sharp daily click drops, though two of five sites examined held steady.

AI & Machine Learning in SEOSEO News & Algorithm UpdatesAnalytics & Measurement
45
News

Graphite estimates AI assistants reached 76 billion monthly sessions worldwide and says it can no longer reliably compare AI search use with search engines.

AI & Machine Learning in SEOAnalytics & MeasurementSEO News & Algorithm Updates
74
Analysis

Google Search Console data shows clicks and click-through rate dropping sharply for many French sites after Google launched AI Overviews and AI Mode in France on July 22, 2026, while some sites were largely unaffected.

AI & Machine Learning in SEOSEO News & Algorithm UpdatesAnalytics & Measurement
25
News

WPP agency Ogilvy launched Echo, an AI tool built with Creative Artists Agency to help marketers meet content needs while protecting consumer trust and talent rights.

AI & Machine Learning in SEOContent & StrategyCareer & Industry Trends