llms.txt: what it is and whether you need one
By Michael Nefedov, noetio · Updated 21 October 2026
llms.txt is a markdown file at the root of your site that gives AI tools a short summary and links to your most useful pages. It is a proposal, not an official standard, and Google says Google Search ignores it.
Short answer
- A proposal from llmstxt.org (v2, modified 10 August 2026), not an official standard.
- Google Search ignores it and says it will “neither harm nor help” visibility. Chrome's Lighthouse has an experimental audit.
- SE Ranking, nearly 300,000 domains: no link to being cited. Cheap to add, but not a priority.
What llms.txt is
llms.txt is one small markdown file that tells an AI tool what your site is and which pages matter.
The idea comes from llmstxt.org. Instead of making an AI tool read your full HTML, you give it one small markdown file with a summary and links to the pages that matter. The proposal says the goal is to give LLM-friendly content, so agents can use a website without parsing everything.
The llms.txt format (the spec)
Only the H1 is required; everything else is optional.
The file is named llms.txt and sits in the site root. The spec allows a subpath too, such as /docs/llms.txt. A file covers the URLs under its path, and where more than one file applies, agents should use the most specific one. The parts, in this order:
| Part | Required? | What it is |
|---|---|---|
| H1 | Yes, the only required part | The name of the site or project |
| Blockquote | No | A short summary |
| Detail text | No | Any markdown except headings |
| H2 sections with link lists | No | Items as [name](url), optionally followed by a colon and notes |
| H2 named “Optional” | No | Links an agent may skip when it needs less context |
The page we read is v2 of the proposal (by Jeremy Howard, published 3 September 2024, modified 10 August 2026). It adds a second idea:
- Pages that agents might need should also offer a clean markdown version at the same URL, with
.mdappended or replacing the extension. - The proposal recommends link relations to help clients find them:
rel="alternate"withtype="text/markdown", andrel="describedby"pointing to the llms.txt file. - The author says llms.txt files are “used most heavily for software documentation”.
This guide covers the llms.txt file, not the markdown pages.
A minimal llms.txt example for a small site
A small site needs one H1, one summary line and a few link lists. This is an invented example for a made-up bakery. Replace the names and links with yours.
# Example Bakery
> Family bakery in Leeds. We sell sourdough, pastries and wedding cakes. Order online for pickup or local delivery.
Opening hours, prices and allergen info are on the pages below. Prices are in GBP.
## Products
- [Sourdough range](https://example-bakery.co.uk/sourdough): Six loaves with ingredients and prices
- [Wedding cakes](https://example-bakery.co.uk/wedding-cakes): Tiers, flavours, lead times and how to order
## Help
- [Delivery and pickup](https://example-bakery.co.uk/delivery): Areas, fees and cut-off times
- [Allergens](https://example-bakery.co.uk/allergens): Allergen table for every product
## Optional
- [Our story](https://example-bakery.co.uk/about): History of the bakery
- [Blog](https://example-bakery.co.uk/blog): Baking tipsLink only pages that answer real questions: what you sell, prices, how to order, policies.
Who reads llms.txt? The evidence
No source shows that Google Search or any major AI system reads the file.
“It's completely fine if you decide to create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files. Doing so will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them.”
“You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them.”
“FWIW no AI system currently uses llms.txt.”
How to weigh these sources
- The Google guide is the main source. It is Google's own documentation and names llms.txt. Its recap says “For Google Search, you can ignore tactics like ‘chunking’ content, creating unnecessary AI text files (like llms.txt), or pursuing inauthentic mentions.” It speaks for Google Search only.
- Mueller's June 2025 post is one person's social post, about 16 months old and about AI systems in general, so we keep it as history.
- Google's older page, AI features and your website (10 December 2025), says you do not need “AI text files” in general terms and does not name llms.txt.
Chrome is a different case
Chrome's Lighthouse tool has an llms.txt audit in its experimental Agentic Browsing category (Chrome docs, updated 5 May 2026).
- Lighthouse flags the page if the server errors when it fetches the file.
- It marks the audit Not Applicable if the file returns a 404, “as providing the file is optional at the moment”.
- The category is marked experimental and has no 0 to 100 score (Chrome docs, scoring). The llmstxt.org page also mentions this audit.
This is a check inside a testing tool, not a statement about Search ranking or AI answers.
SE Ranking's study
~300,000
domains in SE Ranking's dataset. 10.13% had an llms.txt file.
SE Ranking, 15 Oct 2026. Source
No link
between having llms.txt and being cited by AI models. Removing it from their XGBoost model made it more accurate.
SE Ranking. Source
What SE Ranking adds
It says its findings “should be seen as context-dependent”, and its conclusion is: “adding the file is a low-effort way to prepare for the next wave of AI indexing. Today it's optional; tomorrow, it might be essential.” That last line is SE Ranking's opinion, not a result of its data.
AI labs
OpenAI, Anthropic and Perplexity do not mention llms.txt as a crawl control on their bot pages (see the AI crawlers table). The llmstxt.org page says the AI labs publish llms.txt files for their own developer docs. The OpenAI crawlers page and the Perplexity crawlers page both point to their docs index at /llms.txt, and neither says its crawler reads your file. That shows they publish the file, not that their crawlers use yours.
Do you need llms.txt?
Probably not first: it has no proven benefit and no proven harm, so do the basics before it.
- Proven benefit: none documented. Google says Google Search ignores the file.
- Proven harm: none documented. Google says it neither harms nor helps your visibility in Google Search.
- Cost: the time to write and update one text file.
- Worth it? Maybe, if your site is documentation that coding agents read (the spec says that is where it is used most), or you want a clean summary for other tools. Google calls this fine to do for “other services or systems that use these files”. Do it after the basics: allow the right crawlers, get indexed, write clear pages. Google's own view of those basics is in how to rank in AI Overviews.
The same goes for our own site: on 21 October 2026, getcompassseo.com/llms.txt returned 404 when we ran curl -I.
How to add llms.txt in Next.js
Put a static file in public/, or generate it from a route handler if your page list lives in code.
Option 1: a static file
On Next.js (see our Next.js SEO guide), put the file in public/llms.txt. The Next.js docs say files in public are served from the base URL (/), so it loads at /llms.txt. Deploy, then check:
curl -sI https://yoursite.com/llms.txtYou want 200 and a text content type.
Option 2: generate it from your pages
Use a route handler at app/llms.txt/route.ts if your page list lives in code. The Next.js docs show the same pattern for app/rss.xml/route.ts. The @/lib/pages import below is a placeholder for your own list:
import { PAGES } from "@/lib/pages"; // your own list: { href, title, summary }
export function GET() {
const base = "https://yoursite.com";
const lines = [
"# Your Site",
"",
"> One sentence about what you do.",
"",
"## Pages",
"",
...PAGES.map((p) => `- [${p.title}](${base}${p.href}): ${p.summary}`),
];
return new Response(lines.join("\n"), {
headers: { "Content-Type": "text/plain; charset=utf-8" },
});
}Then test it with the same curl command. Do not list pages that are blocked in robots.txt or marked noindex.
llms.txt does not replace robots.txt or content
llms.txt controls no access and earns no ranking. It does not give or deny access. Robots.txt does that. It does not rank you either. What the data says helps is covered in SEO vs GEO. To measure whether AI answers mention you at all, see the LLM SEO tools.
Sources
- The /llms.txt file, v2. Jeremy Howard, llmstxt.org, published 3 September 2024, modified 10 August 2026. https://llmstxt.org/. Read 21 October 2026.
- Optimizing your website for generative AI features on Google Search. Google Search Central, last updated 2026-07-10. https://developers.google.com/search/docs/fundamentals/ai-optimization-guide. Read 21 October 2026.
- AI features and your website. Google Search Central, last updated 2025-12-10. https://developers.google.com/search/docs/appearance/ai-features. Read 21 October 2026.
- llms.txt (Lighthouse audit). Chrome for Developers, last updated 2026-05-05. https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt. Read 21 October 2026.
- Lighthouse agentic browsing scoring. Chrome for Developers. https://developer.chrome.com/docs/lighthouse/agentic-browsing/scoring. Read 21 October 2026.
- Google: no AI system currently uses llms.txt (John Mueller on Bluesky, 17 June 2025). Search Engine Roundtable. https://www.seroundtable.com/google-ai-llms-txt-39607.html. Read 21 October 2026.
- LLMs.txt study, about 300,000 domains. Yulia Deda, SE Ranking, 15 October 2026. https://seranking.com/blog/llms-txt/. Read 21 October 2026.
- Overview of OpenAI Crawlers. OpenAI. https://developers.openai.com/api/docs/bots. Read 21 October 2026.
- Perplexity crawlers. Perplexity docs. https://docs.perplexity.ai/docs/resources/perplexity-crawlers. Read 21 October 2026.
- public Folder. Next.js docs. https://nextjs.org/docs/app/api-reference/file-conventions/public-folder. Read 21 October 2026.
- route.js (Route Handlers). Next.js docs. https://nextjs.org/docs/app/api-reference/file-conventions/route. Read 21 October 2026.
Questions
There is no good evidence that it does. SE Ranking found no link between having the file and being cited, and Google says Google Search ignores it. The spec author says the file is aimed at agents, mostly for software documentation.
No. Robots.txt tells crawlers what they may fetch. llms.txt, as proposed at llmstxt.org, is a markdown overview that points an AI tool to your best pages. It does not block or allow anything.
At the root of your site, so it loads at yoursite.com/llms.txt. The spec also allows the file in a subpath, where it covers the pages under that path.
Not in Google Search, according to Google (quote above). The cost is the time to write the file and keep it current.
Google Search does not. Chrome is a different product: its Lighthouse tool has an experimental llms.txt audit, described in the Chrome section above.
Some sites publish it: SE Ranking found it on 10.13% of nearly 300,000 domains. Publishing is not reading. OpenAI, Anthropic and Perplexity do not say in their bot docs that their crawlers read your file.
More guides
- ClaudeBot: What It Is and How to Block or Allow It
- GPTBot vs OAI-SearchBot vs ChatGPT-User: What to Allow
- AI Crawlers: The Main Bots, robots.txt and Log Checks
- How to Rank in AI Overviews: What Google Says (2026)
- How to Get Cited by ChatGPT: What Is Documented (2026)
Turn your next topic into a useful page
Compass reads your website, helps you plan relevant topics and creates sourced article drafts. Review and approve the work, then send it to a connected CMS or export it.
Written by Michael Nefedov, founder of noetio. Updated 21 October 2026.