Optimixed Search News

Cloudflare adds AI crawler management tool that lets sites set policy by category instead of individual bot

55
News

TLDR

Cloudflare launched Bot Preference Sync, allowing sites to set AI crawler access policies by category rather than managing individual crawlers in robots.txt.

Optimixed Overview

Optimixed's original report on this story.

Cloudflare introduced Bot Preference Sync, a tool that lets sites manage AI crawler access by category type—such as allowing GPTBot while blocking Bytespider—rather than writing individual rules for each crawler in robots.txt. The feature simplifies crawler policy management by grouping bots into categories instead of requiring site owners to maintain separate rules for every individual AI crawler.

Topics:AI & Machine Learning in SEO
Was this useful?
Share

Related articles

35
News

Google Launches Video Series on Tricky Technical SEO Questions

Google launched a Search Central video series answering unconventional technical SEO questions, beginning with topics like emoji, non-English characters, and URL keywords.

Technical SEOSEO News & Algorithm Updates
55
News

Google Adds Soft Navigation Metrics to CrUX Report

Google added experimental soft navigation metrics to the Chrome UX Report to track in-app page transitions, with Search Console support coming next.

Technical SEOSEO News & Algorithm UpdatesTools & Platforms
52
News

Google's Site Migration Processing Can Take 4+ Months

Google's John Mueller clarified that site migrations can take four months or longer to process fully, and offered guidance on what to prioritize during the transition.

Technical SEOSEO News & Algorithm Updates
35
News

HTML Sitemaps Still Have SEO Value, Google Explains

Google clarified that HTML sitemaps can still serve a purpose for SEO despite XML sitemaps becoming the standard.

Technical SEOSEO News & Algorithm Updates