New Study: How Often Do AI Assistants Hallucinate Links? (16 Million URLs Studied)

Posted on September 2, 2025
Source: Data & Studies – SEO Blog by Ahrefs by Ryan Law. Read the original article

TL;DR Summary of AI Assistants Hallucinate URLs and Lead to Higher 404 Rates Than Google

AI assistants such as ChatGPT and Claude frequently generate hallucinated URLs that lead users to non-existent pages, causing higher 404 error rates compared to Google Search. ChatGPT exhibits the highest 404 rate, with 1.01% of clicked URLs and 2.38% of cited URLs resulting in 404s. On average, AI assistants send visitors to 404 pages at almost 3 times the rate of Google. Website owners can identify and mitigate these hallucinated URLs using analytics filtering and redirects.

Optimixed’s Overview: Understanding AI-Generated URL Hallucinations and Their Impact on Website Traffic

What Are Hallucinated URLs and Why Do They Occur?

AI assistants generate URLs that either once existed but now return 404 errors or are entirely fabricated to match expected URL patterns without any real webpage behind them. This behavior arises because AI models rely on a mix of internal knowledge and web search data, which may be outdated or incomplete. As a result, users directed by these AI-generated links often encounter non-existent pages.

Key Findings on 404 Rates from AI Assistants vs. Google

  • ChatGPT leads with the highest 404 rate among AI assistants: 1.01% of clicked URLs and 2.38% of all cited URLs return 404 errors.
  • Other AI assistants like Claude, Copilot, Perplexity, Gemini, and Mistral have lower but still elevated 404 rates compared to Google.
  • The average 404 error rate for AI-referred URLs is approximately 0.43%, nearly 2.87 times greater than Google’s baseline 0.15% 404 rate.
  • 404 rates for Perplexity and Gemini closely align with Google SERP rates, likely because they use Google’s search index, suggesting fewer hallucinations.

How to Detect and Manage Hallucinated URLs on Your Website

Website owners can take proactive steps to identify and reduce the impact of hallucinated URLs:

  • Filter AI traffic in your analytics tools by applying regex filters to isolate visits referred by AI assistants.
  • Use automated scripts or crawlers to check HTTP status codes of URLs cited by AI sources, focusing on those generating 404 errors.
  • Analyze 404 URLs receiving significant traffic (e.g., over 10 visitors per month) to confirm hallucinations.
  • Implement 301 redirects where possible to relevant pages, or enhance your 404 pages with helpful links to retain disappointed visitors.

Why Should Website Owners Care?

Although AI assistant traffic currently represents a small fraction of overall website visits, the presence of hallucinated URLs can degrade user experience and dilute referral traffic quality. Minimizing their impact through simple analytics monitoring and redirects can enhance visitor retention and maintain SEO integrity without significant effort.