Google clarifies that robots.txt governs crawling behavior, not how search engines use content for AI training
TLDR
Google's John Illyes clarified that robots.txt controls crawler access but does not prevent content from being used for AI model training.
Optimixed Overview
Optimixed's original report on this story.
In a new video, Google's John Illyes confirmed that robots.txt is designed to control how search engines crawl and index content, not to restrict how that content may be used for AI training purposes. The clarification addresses a persistent source of confusion among SEOs about whether blocking crawlers via robots.txt also prevents content from being incorporated into AI models. Illyes also recommended using percent-encoding for non-English characters in robots.txt file paths to ensure proper parsing.