llms.txt explained: what it is, and whether it's worth it
TL;DR
- llms.txt is a plain-text file at your site root that points AI systems to your most important, cleanest content.
- It is not robots.txt: robots.txt controls crawling; llms.txt curates and highlights.
- 2026 data is sobering: 97% of llms.txt files get zero AI-crawler requests, and Google says on the record it doesn't use it.
- Keep one because it's cheap and forces useful curation - not because it drives citations. Your AI visibility comes from entity, Bing and citable content.
llms.txt is a simple proposal: a plain-text (Markdown) file at your site root - /llms.txt - that points large language models to your most important, cleanest content, in a format they can read without wading through your navigation, cookie banners and ads.
llms.txt is not robots.txt
People conflate the two because both live at the root and both are about machines. They do opposite jobs. robots.txt tells crawlers what they may or may not fetch - it's about access and restriction. llms.txt assumes access and instead curates: "of everything here, these are the pages that matter, and here's what they're about." One is a fence; the other is a concierge.
How to create yours
The format is deliberately minimal: an H1 with your site or brand name, an optional summary, then Markdown link lists grouped by section.
- Create a file named llms.txt in your site root.
- Start with # Your Brand and a one-line description of what you do.
- Add sections (## Services, ## Key articles) with Markdown links to your best pages.
- Keep it curated - your strongest, most citable pages, not a full sitemap.
- Optionally add an llms-full.txt with expanded content for models that want more.
You can see a working example at miguelmaio.com/llms.txt - and the English version at /en/llms.txt.
Is it worth it today? (the 2026 data)
Here's the honest part most articles skip - with numbers. Adoption sits around 10% of sites and grows slowly. Worse: an Ahrefs study (June 2026, 137,000 domains) found that 97% of llms.txt files received zero requests from AI crawlers - the vast majority are never even read. And Google has said no on the record: in 2025 Gary Illyes confirmed Google doesn't support it and isn't planning to, and John Mueller compared it to the discredited keywords meta tag. No major AI vendor - OpenAI, Anthropic, Google - has committed to using it.
So why keep one? Because the cost is nil, the downside is zero, and it forces a useful exercise: deciding which of your pages are genuinely your best. Where it is reliably read is by coding agents (Cursor, Claude Code, Copilot) on documentation sites. Treat it as a cheap, forward-looking bet - not as what decides your AI visibility. That still comes down to a strong entity, Bing presence and citable content, not a text file. I add it for clients as part of GEO, framed as insurance, not salvation.
Key data
| Files with zero requests | 97% |
|---|---|
| Domains analyzed | 137,210 |
| AI retrieval-bot requests | 1% |
| Google's official position | Doesn't use |
Google confirms it does NOT use llms.txt; no major AI provider has committed to it.
Sources: Ahrefs — llms.txt study · Search Engine Roundtable
Common mistakes and how to do it right
llms.txt is easy to get slightly wrong in ways that make it pointless. From experience:
| Avoid | Do |
|---|---|
| ✗ Treating llms.txt like robots.txt and blocking things | ✓ Using it to highlight your best pages, not to restrict access |
| ✗ Dumping your entire sitemap into it | ✓ Curating only your strongest, most citable pages |
| ✗ Promising clients it guarantees AI citations | ✓ Framing it honestly: a cheap, forward-looking signal, not a switch |
| ✗ Creating it once and letting it rot | ✓ Updating it when your key pages change - it takes a minute |
Tools I use and recommend
- ChatGPTAI assistant for research, analysis and quick audits.
- PerplexityAnswer engine with sources, useful for research and GEO.
- GeminiGoogle's AI assistant, grounded in Search.
- Google Search ConsoleOrganic performance, coverage and field Core Web Vitals.
Sources
- llms.txt specification
- Ahrefs / PPC Land - 97% of llms.txt files get zero AI requests (2026)
- Google Search Central - robots.txt intro
Frequently asked questions
Do AI engines actually read llms.txt?
No vendor guarantees it today. It's an emerging proposal. The value is low-cost, forward-looking positioning, plus the discipline of curating your best pages. GEO guide →
Does llms.txt replace robots.txt or a sitemap?
No. robots.txt controls crawling, a sitemap lists all URLs, llms.txt curates the important ones for AI. They coexist. Technical checklist →