Meet the New Web Crawlers: AI Bots Are Closing in on Search Engine Bots

Posted on June 18, 2025
Source: Data & Studies – SEO Blog by Ahrefs by Patrick Stox. Read the original article

TL;DR Summary of The Growing Impact and Challenges of AI Bots in Web Crawling

The rise of AI bots has significantly increased web crawling activity, now ranking second after search engine bots. While AI bots consume massive resources, their crawling methods are inefficient, causing many websites to block them. Implementing IndexNow can greatly reduce wasted crawl traffic, saving resources and benefiting website owners and the environment. As AI bots evolve, balancing crawl efficiency with providing real value to websites will be essential to avoid further blocking.

Optimixed’s Overview: Understanding the Surge and Efficiency Challenges of AI Web Crawlers

AI Bots Are Reshaping Web Crawling Dynamics

The web crawling landscape is rapidly changing with the emergence of AI bots. Traditionally dominated by search engine crawlers like Googlebot, the space now sees AI bots making up nearly 25% of all bot requests as of May. Despite having fewer numbers, AI crawlers are aggressively expanding their footprint and could surpass search engines in the near future.

Challenges with AI Bot Crawling Efficiency

  • Resource Waste: An estimated 53% of crawler traffic is inefficient or wasted, with AI bots contributing significantly due to their nascent and unrefined crawling strategies.
  • Website Blocking: Many high-traffic sites are increasingly blocking AI bots as a reaction to their aggressive crawling behavior and lack of reciprocal value.
  • Social Contract Importance: Search engines maintain a “social contract” by driving referral traffic and providing transparency to website owners, a balance AI bots must achieve to avoid being blocked.

IndexNow: A Solution for Sustainable and Efficient Crawling

IndexNow emerges as a promising protocol to reduce wasted crawling by notifying search engines and bots about URL changes, dramatically cutting down unnecessary requests. Adoption of IndexNow is encouraged for:

  • Website owners aiming to reduce server load and energy consumption.
  • Content Delivery Networks (CDNs) and Content Management Systems (CMS) to optimize crawl traffic.
  • AI and search engine bots seeking increased crawling efficiency.

Looking Ahead: The Future of AI Crawlers on the Web

As AI bots continue ramping up their crawling activity—exemplified by GPTBot’s recent growth—the need for more refined, efficient crawling practices grows urgent. Providing tangible value to website owners through traffic and data transparency will determine whether AI crawlers are welcomed or blocked. The integration of technologies like IndexNow and the cooperation between AI developers and webmasters will shape the sustainability and impact of AI-driven web crawling.