Google's John Mueller Says Robots.txt Can Block Certain JavaScript Resources From Googlebot
TLDR
Google's John Mueller said it is acceptable, in a specific use case, to use robots.txt to hide certain resources from Googlebot.
Optimixed Overview
Optimixed's original report on this story.
Google's John Mueller said it is okay, within a specific use case, to use robots.txt to keep certain JavaScript resources hidden from Googlebot, as Search Engine Journal reports. The article provides only a brief summary of the guidance, so the exact scenario it covers is not detailed here.
How we got here: earlier coverage
- Oct 22Google Explains What Robots.txt Does and Doesn't Control for AI Content Use
Google clarified that robots.txt controls whether content gets crawled, not how it gets used after crawling, and recommends percent-encoding non-English robots.txt paths.
- Oct 21Google clarifies that robots.txt controls crawler access, not how AI uses indexed data
Google clarified that robots.txt controls which bots can crawl a site, not what those bots do with the data they collect, and that non-standard rules are silently ignored under RFC 9309.
- Oct 21Google Explains What's Actually Supported in robots.txt Files, Including AI Crawler Rules
Google's Gary Illyes and Chery Prommawin released a video explaining which robots.txt directives are actually supported by Google and other search engines or AI crawlers.
- Aug 31John Mueller Shares His Experience Testing Markdown for AI Crawlers
John Mueller tested whether AI crawlers actually request markdown from websites and shared his findings.