OpenAI Claims Robots.txt May Not Stop ChatGPT's Web-Crawling Bot
TLDR
SEOFOMO News reports that OpenAI's documentation suggests ChatGPT's Fetch Bot may not be bound by robots.txt restrictions, despite evidence of it accessing blocked sites.
What this ChatGPT policy change means for your site
- Status
- Policy change
- Platform
- ChatGPT
- Affects
- ChatGPT Fetch Bot, robots.txt compliance
- Who
- Site owners who block AI crawlers with robots.txt
- Effective
- Not specified in the source
- Action needed
- Not specified in the source
Optimixed Overview
Optimixed's original report on this story.
OpenAI's own documentation indicates that ChatGPT's Fetch Bot may not honor robots.txt directives that would normally prevent crawler access. Per SEOFOMO News, new data shows the bot is reaching sites that have explicitly disallowed it, raising questions about whether traditional crawl-blocking methods remain effective against AI training infrastructure.
How we got here: earlier ChatGPT coverage
- Aug 14OpenAI Tests Moving ChatGPT Sources Button Into Hidden Menu
OpenAI is moving the Sources button from ChatGPT responses into a harder-to-find menu, reducing user visibility of source attribution.
- Aug 13Google Search Rankings Volatile, OpenAI Makes Sources Harder to Find
Google experienced a notable ranking volatility spike over August 12–13, while OpenAI began hiding ChatGPT source links behind additional clicks.
- Jul 31OpenAI Rolling Out Business Agent Ads That Launch ChatGPT Conversations Instead of Web Links
OpenAI is testing a new ad format called "Agent" that sends users into ChatGPT business conversations instead of website landing pages, currently available to select advertisers.
- Jul 14OpenAI Clarifies Its Web Crawlers Can Update User-Agent Strings
OpenAI updated its crawler documentation to clarify that its user-agent string version number may change and that it uses an additional robots.txt marker when fetching robots.txt files.