Source: Search Engine Roundtable by barry@rustybrick.com (Barry Schwartz). Read the original article
TL;DR Summary of Should You Add an LLMs.txt File to Your Website?
LLMs.txt is a proposed standard to guide AI systems like ChatGPT in crawling website content, but currently, no AI system uses it. Despite Yoast adding easy generation of this file in their WordPress plugin, adoption remains very low. Google’s John Mueller confirms that major AI chatbots do not fetch or utilize llms.txt files today, relying instead on traditional sitemaps. The future of llms.txt remains uncertain, and its implementation is optional for now.
Optimixed’s Overview: Understanding the Current Role and Relevance of LLMs.txt for SEO
What is LLMs.txt and Why the Buzz?
The LLMs.txt file is a newly proposed standard aimed at helping large language models (LLMs) like ChatGPT better understand and represent website content by providing explicit crawling instructions. It functions similarly to a robots.txt or sitemap file but targets AI content crawlers specifically.
Current Industry Status
- Google’s stance: John Mueller, a prominent Google Search Advocate, publicly stated that no AI systems currently use the
llms.txtfile for crawling or training. - Plugin support: Popular WordPress plugin Yoast has integrated a feature to generate
llms.txtfiles automatically, encouraging site owners to easily provide guidance to LLMs. - Practical adoption: Despite the availability, the file is not yet widely adopted or actively fetched by AI chatbots and models.
SEO and Technical Implications
For now, traditional sitemaps remain the primary tool for guiding crawlers, and having multiple sitemap files from the same CMS does not cause issues. Server logs confirm that LLM-powered chatbots fetch normal pages but ignore llms.txt files.
What Should Website Owners Do?
- Consider experimenting with
llms.txtif you want to be an early adopter, especially if your site uses Yoast or similar tools. - Don’t rely on
llms.txtyet for SEO or AI traffic, as support is not widespread. - Continue maintaining standard SEO best practices, including robust sitemaps and content optimization.
Ultimately, the future usefulness of LLMs.txt depends on AI systems choosing to implement and respect this new standard, so staying informed on updates is recommended.
