Optimixed Search News

Google Explains What's Actually Supported in robots.txt Files, Including AI Crawler Rules

55
News

TLDR

Google's Gary Illyes and Chery Prommawin released a video explaining which robots.txt directives are actually supported by Google and other search engines or AI crawlers.

Optimixed Overview

Optimixed's original report on this story.

Google engineers Gary Illyes and Chery Prommawin released the second video in the Non-Canonical Cabinet series, focusing on robots.txt specifications and what those files can and cannot do. The discussion covers which directives Google supports versus those that other search engines or AI crawlers may not recognize, providing SEO practitioners clarity on proper robots.txt implementation across different crawler types.

Topics:SEO News & Algorithm UpdatesAI & Machine Learning in SEO
Was this useful?
Share

Related articles

45
News

Google Advises On Search Downtime During PHP2Shell Patching

John Mueller provided guidance on maintaining search visibility when temporarily taking a site offline to patch the PHP2Shell vulnerability.

Technical SEOSEO News & Algorithm Updates
28
News

Google Adds Existing Options to Search Data Control Docs

Google updated its help docs to explicitly list existing data-sharing controls including robots meta tags, data-nosnippet, X-Robots-Tag, and Search generative AI controls.

Technical SEOSEO News & Algorithm Updates
42
News

Google Treats Raw and Encoded Brackets as Separate URLs

Google's John Mueller clarified that raw brackets and percent-encoded brackets are treated as separate URLs, advising consistency across links, canonicals, and sitemaps.

Technical SEOSEO News & Algorithm Updates
25
News

Google Updates URL Hyphen Guidance in Search Docs

Google updated its official URL structure documentation to clarify guidance on using hyphens to separate words in URLs.

Technical SEOSEO News & Algorithm Updates