AI Visibility Resource
llms.txt is a Markdown file that sits at the root of your website and gives AI systems a structured summary of your most important content. It was proposed in September 2024 and now sits on roughly one site in ten, but the evidence on whether it actually works is more nuanced than most guides admit.
Check your AI visibilityOr see our full AI optimisation services.
Google rankings and AI citations are two different games. AI platforms like ChatGPT, Perplexity and Google's own AI Overviews don't pull from the same sources as traditional search results. Where Google ranks a list of links, an answer engine retrieves a passage and rewrites it into a single response, so it rewards content it can lift cleanly rather than pages that simply rank well. A page that sits first on Google might never appear in a ChatGPT response.
ChatGPT now has 900 million weekly active users, more than double the 400 million reported in February 2025 (Backlinko, February 2026). Perplexity AI handles more than 100 million queries every week. AI Overviews appear on a growing share of Google searches, and the traffic they absorb would otherwise have gone to your website.
Ahrefs found that while 76% of AI Overview citations come from pages in Google's top 10, a notable 14.4% come from pages that don't rank in the top 100 at all (Ahrefs, 2025). The engines don't just mirror Google; they judge content on their own terms.
Standard web pages are wrapped in navigation menus, JavaScript, CSS, cookie banners and advertising code. When Jeremy Howard released FastHTML in 2024, the top complaint was that AI coding assistants couldn't help developers use it: the documentation existed, but AI systems couldn't extract it efficiently. That's the core problem llms.txt was designed to solve.
If your content isn't structured for AI consumption, that traffic doesn't find you. Traditional SEO optimises for one channel. AI visibility turns on something more specific: whether a crawler can reach the page, whether a retrieval system can isolate the relevant passage, and whether the citation model has a clean enough summary to quote.
How it works
An llms.txt file is a plain Markdown document placed at the root of your website (e.g., https://example.com/llms.txt). It gives AI systems a curated, structured summary of your site: who you are, what you offer, and which pages contain the most useful content.
In practice it is an index for machines: headings that name each section, a one-line description of every important page, and the links that point to them, all in plain Markdown a model can read directly without wading through your navigation and scripts.
Jeremy Howard, co-founder of Answer.AI and fast.ai, published the llms.txt proposal on 3 September 2024. The idea draws an analogy with robots.txt: the decades-old file that tells search crawlers which pages to access. The crucial difference is that instead of issuing permissions, llms.txt is designed to be directly read and understood by language models.
| Feature | robots.txt | llms.txt |
|---|---|---|
| Purpose | Tells crawlers what NOT to access | Tells AI systems what IS most useful |
| Format | Plain text with directives | Markdown with descriptions and links |
| Audience | Search engine crawlers | Language models and AI tools |
| Age | Introduced 1994 (30+ years old) | Proposed September 2024 |
| Adoption | Near-universal across large sites | Early stage |
| Content | Access rules (allow/disallow) | Site summary, key pages, context |
| Effect on AI | Controls crawl access | Provides readable content map |
The file follows a simple Markdown structure: a site name heading, a summary blockquote, grouped page links with descriptions, and an optional section for secondary content. Here's what it looks like for a local business:
What about llms-full.txt? The companion file llms-full.txt contains your complete site content in a single Markdown document: every page, concatenated. Even among sites that have adopted llms.txt, only about 30% also publish llms-full.txt. Some implementations are huge: Vercel's reads like a 400,000-word novel, and Cloudflare's is larger still at 3.7 million tokens (the units of text AI models read). [Publii, 2026]
Identify your 5 to 15 most important pages. These should be the pages you'd want an AI system to reference when someone asks about your business or industry.
Use the Markdown format above. Lead with context. Be specific in your descriptions: "AI visibility audit for UK service businesses" beats "our services."
Upload the file so it's accessible at https://yourdomain.com/llms.txt. Most web hosts and CMS platforms make this straightforward.
Make sure your llms.txt doesn't contradict your robots.txt, sitemap or schema markup. These signals should tell a consistent story.
Review your llms.txt every 3 to 6 months. If your service pages change, your llms.txt should follow.
What the data shows
The evidence is mixed, and anyone telling you llms.txt is a guaranteed visibility boost isn't reading the research. What the studies do show is narrower: a well-structured llms.txt file improves AI visibility compared to an unstructured URL dump, but the structure is doing the work, not the file's mere presence.
SE Ranking analysed nearly 300,000 domains and found just 10.13% had an llms.txt file. Their machine learning model found no correlation between having llms.txt and being cited by AI systems. Removing the llms.txt variable actually improved prediction accuracy.
CiterLabs tracked 1,000 B2B SaaS sites and found that sites with a well-structured llms.txt saw 15 to 40% higher citation rates within 60 days. The key qualifier: most implementations are auto-generated URL dumps, not curated content maps.
In practice, AI crawlers still visit llms.txt files only rarely, and Google has explicitly rejected the standard for AI Overviews. But GPTBot has been observed fetching llms.txt, and Anthropic explicitly recommends it for Claude.
| Metric | Figure | Source |
|---|---|---|
| Top B2B SaaS sites | ~12% (up from <2% mid-2025) | CiterLabs (April 2026) |
| Domains in 300k sample | 10.13% | SE Ranking (November 2025) |
| Inc 5000 companies | <8% | CiterLabs (April 2026) |
| Factor | Without llms.txt | With llms.txt |
|---|---|---|
| AI content access | AI crawlers read raw HTML (navigation, ads, scripts included) | AI systems receive curated Markdown (clean, structured) |
| Content priority | AI must guess which pages matter | Site owner explicitly flags key pages |
| Context efficiency | Tokens wasted on non-content elements | Tokens focused on useful information |
| Implementation effort | None | 30 to 60 minutes for a basic file |
| Guaranteed AI citation? | No | No: but reduces friction for AI systems that do check |
| Risk | None | Minimal (can expose site structure publicly) |
weekly active users on ChatGPT alone, more than double the figure from early 2025. AI-driven search traffic is growing at a pace that makes preparing for it a matter of when, not whether. [Backlinko, February 2026]
llms.txt is one signal in a larger system. The research points to several factors that carry more weight:
llms.txt supports all of this. It doesn't replace it. For a broader self-assessment of your AI readiness, see the AI visibility checklist for local businesses.
Not yet. It's a community-driven proposal, first published by Jeremy Howard in September 2024. It has reached what CiterLabs describes as "working-standard status" by April 2026, with endorsement from Anthropic and adoption by major tech companies. But it has no W3C or IETF backing, and Google has explicitly declined to use it for AI Overviews.
No file can force any AI platform to recommend your business. A well-structured llms.txt makes your site's most useful content easier for AI systems to find and read. Whether those systems choose to cite you depends on content quality, domain authority, relevance and platform-specific behaviour: factors llms.txt doesn't control.
No. Google has stated that AI Overviews and AI Mode rely on traditional SEO signals. Google's position is that good content, structured data and technical health are what matter for AI search visibility within their ecosystem. That said, ChatGPT, which drives a large share of AI referral traffic, operates independently of Google's stance.
It depends on your site. If you have multiple service pages, resources and proof content, a short llms.txt that points to your strongest material takes 30 to 60 minutes and carries no technical risk. If your site is thin (only a few pages with limited content) fix the content first. llms.txt works best when it points to pages worth reading.
You can, but be careful. Yoast and Rank Math both offer auto-generation features. The problem: most auto-generated files are URL dumps with no descriptions or context. SE Ranking's 300,000-domain study didn't differentiate between well-crafted and auto-generated files, and CiterLabs found the quality of implementation matters. A hand-written file with real descriptions outperforms an auto-generated list every time.
Review yours every 3 to 6 months. At minimum, update it whenever you add, remove or significantly change a key page. An outdated llms.txt that links to dead pages or obsolete services does more harm than good.
An llms.txt file only helps if it points to pages worth reading. Point it at thin or vague service pages and it just indexes the problem. So before you write one, it is worth knowing whether your core pages are clear, crawlable and backed by valid schema. The free check tells you that.
You get a plain read on which pages are ready to be cited and which need work first. No promises about rankings; nobody controls the platforms.