Cloudflare To Block AI Crawlers By Default

Posted on July 1, 2025
Source: Search Engine Roundtable by barry@rustybrick.com (Barry Schwartz). Read the original article

TL;DR Summary of Cloudflare Blocks AI Crawlers by Default and Introduces Pay Per Crawl

Cloudflare now blocks AI crawlers by default, shifting control to content creators. They launched Pay Per Crawl, a system allowing publishers to monetize AI access to their content. Publishers can fully customize crawler permissions, enabling a fair compensation model for content use. This move impacts about 20% of the web and major publishers are already participating.

Optimixed’s Overview: How Cloudflare Empowers Publishers with AI Crawler Control and Monetization

Cloudflare’s New Default AI Crawler Blocking

Cloudflare, which supports roughly 20% of all websites, has implemented a groundbreaking change by blocking AI crawlers from accessing web content by default. This policy applies automatically to new users, while existing customers can enable blocking with a simple dashboard toggle. This shift moves web scraping controls from an opt-out system to an opt-in framework, giving content creators stronger authority over their material.

Introducing Pay Per Crawl: Monetizing AI Access

Alongside blocking AI crawlers, Cloudflare unveiled Pay Per Crawl, an innovative model that permits publishers to set fees for AI services wishing to crawl their content. This initiative allows for:

  • Full control over crawler access—block all, allow select crawlers, charge for access, or offer it for free.
  • AI companies to review pricing and decide whether to pay for crawling rights.
  • Content creators to monetize their intellectual property fairly in the AI age.

Major Industry Participation and Future Impact

Notable publishers such as BuzzFeed, The Atlantic, TIME, Stack Overflow, and O’Reilly Media have joined this initiative, signaling broad industry support. By empowering creators with both protection and monetization options, Cloudflare is pioneering a sustainable ecosystem for AI content usage. This development is poised to reshape how AI services interact with web content and compensate originators moving forward.