Out of Cite: Why Your PR Wins May Be Missing From AI Search – Screaming Frog

Posted on July 22, 2025
Source: Screaming Frog by Chris Jones. Read the original article

TL;DR Summary of How News Sites Blocking AI Crawlers Impact Digital PR and Brand Visibility

Large Language Models (LLMs) like ChatGPT are reshaping search marketing by influencing brand visibility through AI-powered search. However, 75% of major news sites block AI crawler bots (especially OpenAI’s GPTBot), limiting LLMs’ ability to access and cite their content. This creates a challenge for Digital PR as press coverage on blocked sites becomes invisible to AI-driven searches. To maintain brand authority, PR strategies must now focus on securing coverage on sites that allow AI crawling while continuing traditional outreach.

Optimixed’s Overview: Navigating the New Digital PR Landscape Amidst AI Crawler Restrictions

The Rise of AI-Powered Search and Its Impact on Brand Visibility

With AI tools like ChatGPT, Gemini, and DeepSeek changing how people find and trust information online, Large Language Models (LLMs) have become key players in search marketing. Unlike traditional search engines that rank pages based on links and keywords, these models generate synthesized, conversational answers drawing from vast datasets, including news content.

This shift means brands must be visible not only to humans but also to AI systems that influence consumer decisions. However, a major challenge has emerged: many prominent news publishers are blocking AI crawler bots from indexing their content, affecting how LLMs access and cite media coverage.

Key Findings from the Study on AI Crawler Access

  • 75% of major news sites surveyed block at least one AI crawler bot, with GPTBot (OpenAI’s crawler) being the most frequently blocked (58%).
  • Leading outlets like MailOnline, iNews, Metro, and BBC News block nearly all AI bots (91%), severely limiting AI access to their content.
  • Regional news sites block more AI crawlers on average (8) compared to national (5) and consumer news sites (3).
  • Despite widespread blocking, about 25% of news sites allow all major AI crawlers, offering valuable opportunities for Digital PR to secure AI-visible coverage.

Implications for Digital PR and Brand Strategy

The blocking of AI crawlers by many news publishers stems from concerns over copyright infringement, unlicensed use of content by AI companies, and revenue loss due to “zero-click searches.” This evolving dynamic means:

  • AI visibility is now crucial for offsite SEO and Digital PR success, requiring brands to target news outlets that permit AI crawling.
  • Traditional PR goals remain relevant, but strategies must adapt to serve dual audiences: human readers and AI algorithms.
  • PR measurement should incorporate LLM visibility metrics alongside conventional coverage metrics to capture brand presence in AI-driven search results.
  • Ongoing legal and commercial negotiations between media owners and AI firms will continue to shape content accessibility.

Recommendations for Digital PR Professionals

To effectively navigate the AI-influenced search ecosystem, PR teams should:

  • Understand which news sites allow or block AI crawler bots and prioritize outreach accordingly.
  • Maintain a balanced content strategy that includes authoritative brand-owned content and earned media placements on AI-accessible platforms.
  • Monitor developments around AI content licensing and crawler policies to anticipate changes impacting brand visibility.
  • Educate clients on the importance of AI search visibility and update reporting to reflect LLM citation potential.

In this rapidly evolving landscape, successful Digital PR will hinge on agility, technical awareness, and strategic targeting of both human and machine audiences to maximize brand reach and influence.