Optimixed Search News

OpenAI Claims Robots.txt May Not Stop ChatGPT's Web-Crawling Bot

By Eric Mandell
72
News

TLDR

SEOFOMO News reports that OpenAI's documentation suggests ChatGPT's Fetch Bot may not be bound by robots.txt restrictions, despite evidence of it accessing blocked sites.

What this ChatGPT policy change means for your site

Status
Policy change
Platform
ChatGPT
Affects
ChatGPT Fetch Bot, robots.txt compliance
Who
Site owners who block AI crawlers with robots.txt
Effective
Not specified in the source
Action needed
Not specified in the source

Optimixed Overview

Optimixed's original report on this story.

OpenAI's own documentation indicates that ChatGPT's Fetch Bot may not honor robots.txt directives that would normally prevent crawler access. Per SEOFOMO News, new data shows the bot is reaching sites that have explicitly disallowed it, raising questions about whether traditional crawl-blocking methods remain effective against AI training infrastructure.

How we got here: earlier ChatGPT coverage

  1. Aug 14
    OpenAI Tests Moving ChatGPT Sources Button Into Hidden Menu

    OpenAI is moving the Sources button from ChatGPT responses into a harder-to-find menu, reducing user visibility of source attribution.

  2. Aug 13
    Google Search Rankings Volatile, OpenAI Makes Sources Harder to Find

    Google experienced a notable ranking volatility spike over August 12–13, while OpenAI began hiding ChatGPT source links behind additional clicks.

  3. Jul 31
    OpenAI Rolling Out Business Agent Ads That Launch ChatGPT Conversations Instead of Web Links

    OpenAI is testing a new ad format called "Agent" that sends users into ChatGPT business conversations instead of website landing pages, currently available to select advertisers.

  4. Jul 14
    OpenAI Clarifies Its Web Crawlers Can Update User-Agent Strings

    OpenAI updated its crawler documentation to clarify that its user-agent string version number may change and that it uses an additional robots.txt marker when fetching robots.txt files.

Was this useful?
Share

More AI & Machine Learning in SEO stories

62
News

An analysis of Search Console data from French sites links AI Overviews to sharp daily click drops, though two of five sites examined held steady.

AI & Machine Learning in SEOSEO News & Algorithm UpdatesAnalytics & Measurement
45
News

Graphite estimates AI assistants reached 76 billion monthly sessions worldwide and says it can no longer reliably compare AI search use with search engines.

AI & Machine Learning in SEOAnalytics & MeasurementSEO News & Algorithm Updates
74
Analysis

Google Search Console data shows clicks and click-through rate dropping sharply for many French sites after Google launched AI Overviews and AI Mode in France on July 22, 2026, while some sites were largely unaffected.

AI & Machine Learning in SEOSEO News & Algorithm UpdatesAnalytics & Measurement
25
News

WPP agency Ogilvy launched Echo, an AI tool built with Creative Artists Agency to help marketers meet content needs while protecting consumer trust and talent rights.

AI & Machine Learning in SEOContent & StrategyCareer & Industry Trends