Optimixed Search News

Google clarifies that robots.txt governs crawling behavior, not how search engines use content for AI training

72
News

TLDR

Google's John Illyes clarified that robots.txt controls crawler access but does not prevent content from being used for AI model training.

Optimixed Overview

Optimixed's original report on this story.

In a new video, Google's John Illyes confirmed that robots.txt is designed to control how search engines crawl and index content, not to restrict how that content may be used for AI training purposes. The clarification addresses a persistent source of confusion among SEOs about whether blocking crawlers via robots.txt also prevents content from being incorporated into AI models. Illyes also recommended using percent-encoding for non-English characters in robots.txt file paths to ensure proper parsing.

Was this useful?
Share

Related articles

45
News

Publishers are coordinating across the US and EU to push for legislation requiring AI bots to identify themselves as a defense against scraping.

SEO News & Algorithm UpdatesAI & Machine Learning in SEOContent & Strategy
52
News

Google expanded its list of content exclusion methods to seven, adding Search Console AI controls alongside existing snippet and indexing rules.

SEO News & Algorithm UpdatesAI & Machine Learning in SEOTools & Platforms
65
News

Google updated its URL best practices documentation as the search community reported elevated ranking volatility over the past day.

SEO News & Algorithm UpdatesTechnical SEO
65
News

New details about Google Discover's SDF (Source Document Freshness) algorithm guide publishers on content recency and mix strategy for better visibility.

SEO News & Algorithm UpdatesContent & Strategy