Optimixed Search News

OpenAI Claims Robots.txt May Not Stop ChatGPT's Web-Crawling Bot

72
News

TLDR

SEOFOMO News reports that OpenAI's documentation suggests ChatGPT's Fetch Bot may not be bound by robots.txt restrictions, despite evidence of it accessing blocked sites.

Optimixed Overview

Optimixed's original report on this story.

OpenAI's own documentation indicates that ChatGPT's Fetch Bot may not honor robots.txt directives that would normally prevent crawler access. Per SEOFOMO News, new data shows the bot is reaching sites that have explicitly disallowed it, raising questions about whether traditional crawl-blocking methods remain effective against AI training infrastructure.

Was this useful?
Share

Related articles

62
News

Meta's Muse AI bot generated one-third of all AI browser traffic in its first 9 days, mostly from CDN infrastructure rather than consumer devices, per security firm DataDome.

AI & Machine Learning in SEOTechnical SEOTools & Platforms
52
News

Google has again adjusted how links appear in AI Mode and AI Overviews, continuing efforts to drive clicks from AI responses to publisher sites.

AI & Machine Learning in SEOSEO News & Algorithm Updates
75
Analysis

Analysis suggests Google will shift its default from ranked search results to AI-generated answers once ad revenue from AI answer pages matches traditional SERP yields, a transition expected between 2028 and 2030.

AI & Machine Learning in SEOSEO News & Algorithm Updates
55
Research

SE Ranking's analysis of 300,000 domains found limited evidence that LLMs.txt files materially affect AI citation rates.

AI & Machine Learning in SEOAnalytics & MeasurementTools & Platforms