Optimixed Search News

OpenAI Clarifies Its Web Crawlers Can Update User-Agent Strings

32
News

TLDR

OpenAI updated its crawler documentation to clarify that its user-agent string version number may change and that it uses an additional robots.txt marker when fetching robots.txt files.

Optimixed Overview

Optimixed's original report on this story.

SERoundtable reports that OpenAI made clarification changes to its crawler help documentation, noting that the version number of the user-agent string may be updated and that OpenAI may use a user-agent string with an additional robots.txt marker to help site owners distinguish requests for robots.txt files in their logs. Barry Schwartz notes that while the updates themselves are not particularly novel, the documentation changes were worth highlighting to the SEO community.

Topics:SEO News & Algorithm UpdatesAI & Machine Learning in SEO
Was this useful?
Share

Related articles

55
Research

49% of Marketers Plan to Deprioritize SEO, Survey Shows

A Typeface survey of 500+ marketing leaders found 49% plan to deprioritize SEO, with 62% reporting brands sacrifice content quality for AI citations.

SEO News & Algorithm UpdatesAI & Machine Learning in SEOContent & Strategy
68
News

Google Raises Inline Links in AI Overviews, Expands Preferred Sources

Google is repositioning inline links higher in AI Overviews and expanding carousels for Preferred Sources.

SEO News & Algorithm UpdatesAI & Machine Learning in SEO
55
News

Panda-Style Google Update Predicted Within Month

A prediction model suggests a 96.8% probability that Google will roll out a Panda-style update targeting scaled AI content and fake author information within the next month.

SEO News & Algorithm UpdatesAI & Machine Learning in SEOContent & Strategy
25
News

Google Tests Redesigned Search Bar in Mobile App

Google is testing a redesigned search bar in its mobile app with a new 'Ask Google' label and visual updates.

SEO News & Algorithm Updates