Llms.txt is a proposed web standard that gives AI systems like ChatGPT a curated guide to your website’s best content
Created by Jeremy Howard in September 2024, it uses simple Markdown format and lives at yoursite.com/llms.txt
No major AI company has confirmed they actually use these files when crawling websites
Research analysing 300,000 domains found no measurable impact on AI citations
For most small business websites, proven methods like schema markup and clear content structure deliver better results
The llms.txt file is a proposed web standard that helps AI systems like ChatGPT find your website’s best content. AI platforms now answer millions of questions by pulling information from websites, but these systems weren’t built to browse the web like humans do.
They struggle with navigation menus, JavaScript, adverts, and all the visual clutter that makes websites work for people.
A proposed solution called llms.txt has sparked debate in SEO circles. You may have seen it mentioned online or wondered whether your competitors are already using it. The claims vary wildly.
Some call it essential for AI visibility. Others dismiss it as another overhyped web standard that will never gain traction.
This guide cuts through the noise. You’ll learn what llms.txt actually is, whether AI systems use it, and whether adding one to your site is worth the effort. No marketing spin, just practical information to help you decide.
Table of Contents
What Is Llms.txt?
Llms.txt is a text file you add to your website to help large language models understand your website content.
Jeremy Howard, co-founder of Answer.AI, proposed it in September 2024. The concept is simple: instead of letting AI systems figure out your website on their own, you hand them a curated list of your most useful pages.
Think of it as creating a reading list specifically for AI assistants. Rather than having ChatGPT or Claude wade through your entire site, you point them straight to the content that matters.
How Does Llms.txt Differ from Robots.txt and Sitemaps?
Your website probably already has robots.txt and an XML sitemap. Each serves a different purpose:
Robots.txt tells search engine crawlers which pages they can and cannot access. It sets boundaries.
XML sitemaps list all your indexable pages so search engines can find them. They provide a complete inventory.
Llms.txt does neither. It highlights your best content and explains what each page contains. Think of it as a curated guide rather than a comprehensive catalogue or a set of rules.
How Does Llms.txt Work?
The txt file sits in your website’s root directory at yoursite.com/llms.txt. Unlike XML sitemaps that use structured markup, llms.txt uses Markdown format because AI systems understand natural language better than code.
When an AI system visits your site, it can look for this file and quickly understand your content structure.
A basic llms.txt file follows a specific format:
- A heading with your company or project name
- A brief summary explaining what you do
- Optional notes about your content
- Organised sections linking to your key pages
What Goes in an Llms.txt File?
Here’s what the format looks like in practice:
# Smith's Plumbing Services
> Family-run plumbing company serving Greater Manchester since 1985. Emergency callouts, installations, and repairs for homes and businesses.
We specialise in boiler repairs and bathroom installations. All work guaranteed for 12 months.
## Our Services
- [Emergency Plumbing](https://smithsplumbing.co.uk/emergency): 24/7 callout service for burst pipes, leaks, and boiler breakdowns
- [Bathroom Installations](https://smithsplumbing.co.uk/bathrooms): Complete bathroom fitting from design to completion
- [Boiler Services](https://smithsplumbing.co.uk/boilers): Gas Safe registered engineers for repairs and installations
## Helpful Guides
- [Common Boiler Problems](https://smithsplumbing.co.uk/blog/boiler-problems): Troubleshooting guide for homeowners
- [FAQ](https://smithsplumbing.co.uk/faq): Answers to common plumbing questionsThe proposal also suggests creating Markdown versions of your web pages by adding .md to URLs. This gives AI systems clean text without navigation clutter.
Do AI Crawlers Actually Use Llms.txt?
Well.
Google’s John Mueller compared it to the keywords meta tag, which search engines abandoned years ago. Gary Illyes from Google stated they don’t support it and have no plans to.
When Semrush tested llms.txt on Search Engine Land over several months (mid-August to late October 2025), they found zero visits to the file from GPTBot, PerplexityBot, or ClaudeBot. Traditional search crawlers like Googlebot occasionally checked the file, but AI-specific crawlers showed no interest.
SE Ranking analysed 300,000 domains and found only 10.13% had implemented llms.txt. More significantly, they found no correlation between having the file and being cited by AI systems. Their machine learning model actually performed better at predicting AI citations when the llms.txt variable was removed entirely.
Google’s Gary Illyes addressed llms.txt directly at the Google Search Central Deep Dive event in July 2025, stating that “Google doesn’t support llms.txt and isn’t planning to” and emphasising that normal SEO practices are sufficient for ranking in AI Overviews.
Gary Illyes, Google Search Relations Engineer
Some interesting signals do exist, though. Anthropic (the company behind Claude) has published an llms.txt file on their own documentation site. Developer-focused companies like Cloudflare, Vercel, and Stripe have implemented it too.
Around 780 to 950 websites now use the format, but that’s a tiny fraction of the web. According to Rankability’s scan of the top 1,000 most-visited websites globally, not a single major platform has implemented llms.txt.
Should You Add Llms.txt to Your Website?
The answer depends on what you’re hoping to achieve and how much time you have.
When Llms.txt Makes Sense
Developer tools and technical documentation benefit most. If your audience includes developers using AI coding assistants like Cursor or GitHub Copilot, structured documentation helps those tools give accurate suggestions.
Future-proofing with minimal effort applies if you can create the file fairly easily. If AI platforms eventually adopt the standard, you’ll already be set up.
Complex websites with deep content where AI might miss your best material could benefit from explicit guidance, though the evidence for this remains thin.
When You Can Skip It
Most small business service websites won’t see measurable benefits from llms.txt right now. Your time delivers better returns when spent on proven tactics like schema markup and clear content structure.
Small sites with limited content don’t need a curated guide. AI systems can easily process a website with 20 or 30 pages without help.
When you have bigger priorities, focus on content quality, E-E-A-T signals, and answering customer questions directly. These factors have a documented impact on AI citations.
How Do You Create an Llms.txt File?
If you decide to add llms.txt to your site, the process is straightforward.
Creating it manually works for most websites. The official specification at llmstxt.org explains the format in detail. Open any text editor, format your content using the Markdown structure shown above, save it as llms.txt, and upload it to your website’s root directory via FTP or your hosting file manager.
WordPress users have a few plugin options. Yoast SEO and Rank Math both now include llms.txt generation. A dedicated plugin called Website LLMs.txt (with over 30,000 installs) can auto-generate the file and track whether AI crawlers access it.
Keep your file simple. List your 20 to 50 most important pages rather than everything on your site. Write clear descriptions explaining what each page covers. Update the file every few months when you publish significant new content or remove old pages.
What Actually Improves AI Visibility?
While llms.txt remains unproven, other approaches deliver measurable results for AI search visibility.
Content structure matters more than most people realise. Break your pages into clear sections that each answer a specific question. AI systems extract information in chunks, so make each section stand alone.
Schema markup makes a genuine difference. Pages with well-implemented schema are more likely to appear in AI Overviews and be cited by AI systems. FAQ schema, Article schema, and LocalBusiness schema are particularly effective.
E-E-A-T signals influence what AI systems cite. Display your expertise and credentials clearly. Include author bios with relevant qualifications. AI platforms prioritise content from recognised authorities.
Question-based headings match how people query AI. Format your H2 and H3 headings as questions your customers actually ask. “How much does a boiler service cost?” works better than “Boiler Service Pricing” for AI visibility.
These approaches have documented evidence behind them. They’re where your time and effort will pay off.
Making Your Decision
Llms.txt is an interesting idea that hasn’t achieved the adoption needed to matter yet. The concept makes sense: give AI systems clear guidance about your content. Implementation is simple enough that it costs little time.
But the evidence shows it doesn’t currently affect whether AI platforms cite your website.
The major players haven’t committed to using it. Server logs from publishers who’ve tested it show AI crawlers ignoring the file entirely.
For most UK small businesses, your priorities should be content quality, proper schema markup, clear site structure, and demonstrating your expertise. These proven approaches will serve you better than chasing an unproven standard.
If you have a spare hour and fancy experimenting, adding llms.txt won’t hurt.
Just don’t expect it to transform your AI visibility overnight. The way platforms interact with websites is still evolving, and the format that eventually wins might look quite different.
What works regardless of technical standards is creating genuinely helpful content that answers your customers’ questions clearly. That serves both human visitors and AI systems well, no matter which specifications gain traction.
Frequently Asked Questions
Llms.txt is a small text file you add to your website that tells AI systems like ChatGPT which pages and articles matter most. Instead of letting AI crawl your entire site and guess what’s important, you provide a curated list with descriptions. The file uses simple Markdown format and lives at yoursite.com/llms.txt.
No. They serve opposite purposes. Robots.txt tells search engines which pages they cannot access. Llms.txt highlights which pages AI should pay attention to. Think of robots.txt as a “keep out” sign and llms.txt as a welcome guide pointing visitors to your best rooms.
No major AI company has confirmed they use these files. Server log analysis from Semrush showed AI crawlers like GPTBot and ClaudeBot made zero visits to llms.txt files over several months of testing. Google has explicitly stated they don’t support the standard.
Probably not as a priority. Current research shows no measurable benefit for AI citations. Your time delivers better results when spent on schema markup, clear content structure, and demonstrating your expertise. If you have a spare hour and want to experiment, there’s no harm in adding it.
Between 30 minutes and two hours for most websites. You need to identify your most important pages, write brief descriptions for each, and format everything in Markdown. WordPress plugins can automate much of this if you’d rather not do it manually.
Yes. Yoast SEO and Rank Math SEO now include llms.txt generation. Dedicated plugins like Website LLMs.txt can automatically create and maintain the file, plus track whether any AI crawlers actually access it. This takes the manual work out of implementation.
An XML sitemap lists every indexable page on your website so search engines can find them all. Llms.txt is selective. You choose only your best 20 to 50 pages and add descriptions explaining what each one covers. Sitemaps are comprehensive catalogues. Llms.txt files are curated highlights.
No. Google representatives have explicitly stated they don’t support llms.txt and aren’t planning to. John Mueller compared it to the deprecated keywords meta tag that search engines stopped using years ago. Other AI platforms might adopt it eventually, but Google’s position is clear.
Current research suggests it won’t. A study of 300,000 domains found no correlation between having llms.txt and being cited in AI-generated responses. Content quality, clear structure, authority signals, and schema markup have far greater influence on whether AI systems reference your site.
Proven tactics include implementing schema markup (especially FAQ and Article schema), using question-based headings that match how people query AI, displaying author credentials clearly, including statistics with sources, and writing content that directly answers common questions. These approaches have documented evidence supporting their effectiveness.
Probably not yet. These pages help AI tools like ChatGPT find accurate information about your brand, but they work best for businesses already being mentioned by AI assistants or those with complex offerings AI might misrepresent.
For most small business websites, your time delivers better results when spent on clear content structure, schema markup, and helpful information that answers customer questions. If AI tools are already getting facts wrong about your business, an AI info page becomes more worthwhile.