llms.txt: What the Spec Says, Who Reads It, and What Mine Looks Like

macmdviewer.com serves an llms.txt. I wrote it by hand: 14,243 bytes, twelve ## sections, ninety curated links. As far as I can measure, it has bought me nothing in AI search. I still keep it, which is not the position most articles about this file will sell you.

So here is the honest version. An llms.txt file is a markdown file at the root of a domain that hands language models a curated index of the site. The spec is real and precise, adoption is growing fast, and the 2026 research says almost nobody reads these files. All three are true at once. I build MacMD Viewer, a $19.99 one-time markdown reader for macOS, which is the only reason the product appears here at all: an llms.txt is markdown, and checking one in a browser tab is genuinely unpleasant.

TL;DR: llms.txt is a markdown file at your domain root, proposed by Jeremy Howard in September 2024. The H1 is its only required element. Google Search ignores it and says so in writing. No major AI provider documents support for it. The one audience that demonstrably fetches it is coding agents, and only when a developer points them at it.

Everything below was checked against primary sources on 26 July 2026. This topic moves fast enough that dates matter, so every claim carries one.

What Is an llms.txt File?

It is a markdown file at /llms.txt that tells a language model what your site contains and where the good parts are. The spec, published by Answer.AI and fast.ai co-founder Jeremy Howard on 3 September 2024, states the purpose in one line: "A proposal to standardise on using an /llms.txt file to provide information to help LLMs use a website at inference time."

The format choice is deliberate. The spec is explicit that it "uses Markdown to structure the information rather than a classic structured format such as XML" because "we expect many of these files to be read by language models and agents." That decision is why the file is human-readable at all, and why checking one comes down to opening markdown.

One thing it is not, despite the filename: a directive. Ahrefs put it exactly right in its 2026 study — it "controls nothing and blocks nothing." No allow, no deny, no crawl-delay. Any article promising control over how AI crawlers reach your content through llms.txt has the mechanism wrong.

What Does the llms.txt Spec Actually Require?

One H1. Everything else is optional, which surprises most people who have read a guide about it.

The spec lists the sections a conforming file contains, "as markdown, in the specific order":

• an optional byte-order mark
• an H1 with the name of the project or site, "the only required section"
• a blockquote with a short summary of the project
• zero or more markdown sections of any type except headings, giving more detail
• zero or more sections delimited by H2 headers, containing file lists of URLs

Each file list is a markdown list whose items carry a required hyperlink, then optionally a colon and notes. Here is the official mock example, verbatim from the spec:

# Title
 
> Optional description goes here
 
Optional details go here
 
## Section name
 
- [Link title](https://link_url): Optional link details
 
## Optional
 
- [Link title](https://link_url)

Two details are worth reading twice, because most guides get them wrong. First, that no-headings rule applies to the prose between the blockquote and your first ##. Second, ## Optional is a reserved section name, not a suggestion: "if it's included, the URLs provided there can be skipped if a shorter context is needed." Put secondary material there, not your best pages. For the underlying link-list and blockquote syntax, my markdown cheat sheet covers it.

Is llms-full.txt Part of the Spec?

No, and this is the single most repeated error on the topic. The llmstxt.org spec never defines llms-full.txt. The companion files it does describe come from FastHTML and are named differently: "The two files are: llms-ctx.txt, which does not include the optional URLs, and llms-ctx-full.txt, which does include them."

llms-full.txt, meaning an entire documentation site concatenated into one file, is a tooling convention popularised chiefly by Mintlify, which documents it as its own feature. Publishing one does not follow from publishing the other either: docs.stripe.com/llms.txt exists, and docs.stripe.com/llms-full.txt returned 404 when I checked on 26 July 2026.

The sizes are the interesting part. Fetched the same day, Anthropic's llms-full.txt was 25 MB and Cloudflare's 56 MB. A 56 MB context file vastly exceeds any current model's context window, which undercuts the usual "just feed it to the LLM" pitch.

Adoption: Two Studies, Two Very Different Numbers

Two teams measured this in June 2026 and got 28% and 7.4%. Both are right, and the gap is the most useful thing in the data.

Ahrefs checked all 137,210 domains in its Web Analytics data that received traffic in May 2026, keeping only files that were real markdown rather than HTML (Ahrefs, 15 June 2026). It found 28% publish an llms.txt, with a candid caveat attached: its customers "skew more technical and SEO-aware than the web at large, so treat the 28% adoption figure as an upper bound."

Chris Humphrey fetched /llms.txt for the top 10,000 Majestic Million domains, followed redirects, and kept only genuine markdown (Humphrey, 18 June 2026). Result: 7.4%, or 737 sites. Two further findings there matter more than the headline. Of 1,050 domains returning HTTP 200, 313 were serving an HTML soft 404 rather than a file. And of the 737 real files, only 55% had the recommended shape of title, summary, and at least one section of curated links.

Humphrey's summary is the line to remember: "There is no single universal adoption rate, only the rate inside a defined slice." Adoption is flat by rank in his data too, leading at 11% in the top 100 then wobbling between 6% and 9% all the way down.

Some live files I fetched on 26 July 2026:

Both platform.openai.com/llms.txt and huggingface.co/llms.txt returned 404 the same day, the latter despite still appearing in a public llms.txt directory. Those directories go stale, which is why I would not use one as an adoption measure.

Does llms.txt Help You Rank or Get Cited by AI?

No, on the evidence available in July 2026. This is the part most articles on the topic get wrong, so the primary sources are worth quoting rather than summarising.

Google's AI optimization guide, last updated 10 July 2026, is unambiguous: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them." It goes on: "Doing so will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them."

Search Advocate John Mueller made the point earlier and less diplomatically, in an April 2025 Reddit thread reported by Search Engine Journal: "AFAIK none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it). To me, it's comparable to the keywords meta tag – this is what a site-owner claims their site is about …"

The Ahrefs request data backs that up hard. Of roughly 38,000 valid llms.txt files, 97% received zero requests in May 2026 — no bots, no humans. Where traffic did arrive, 96% of it was bots, and 77% of those bots were not AI tools at all, led by SEO audit crawlers at 21.7%. Named AI bots made up 19.5%, which Ahrefs stresses is a ceiling because "'fetched' doesn't mean 'read'." Inside that slice, AI retrieval bots, the ones that would actually drive AI-search citations, were 1.1%. Slackbot fetched llms.txt files more often than PerplexityBot did. And AI bots made zero requests for files that do not exist: "AI tools never go looking for llms.txt files that aren't there, so publishing one does not put you on any AI radar."

The strongest good-faith rebuttal comes from Carolyn Shelby, SEO and AI Strategist at Yoast, in Search Engine Land, July 2025. The meta keywords analogy fails on verifiability, she argues: that tag "made unverifiable claims," whereas llms.txt "curates a list of real URLs, and the content has to exist — and deliver — when the model gets there." On necessity she is blunt: "robots.txt isn't 'necessary' either, and neither is schema." Fair points about mechanism, and neither is evidence of a citation lift. No such evidence exists yet.

Then came the week in late May 2026 when Google took both sides. Search Central published the mythbusting guidance above; days later the Chrome team shipped an llms.txt check inside Lighthouse's Agentic Browsing audits, reasoning that "without this file, agents may spend more time crawling the site to understand its high-level structure and primary content." Since I have seen this misreported: the audit fails only on a server error, and a missing file is marked Not Applicable, "as providing the file is optional at the moment." Lighthouse requires nothing and penalises nothing. The "at the moment" is Chrome's phrasing, not mine.

Mueller supplied the reconciliation himself, in comments reported by Ahrefs: llms.txt is "not done for search," and is closer to a "temporary crutch, perhaps to save some tokens" for AI coding tools reading developer documentation. Two Google surfaces, two answers. Search ignores it; agents may use it.

So Who Actually Reads These Files?

Coding agents, and only when you point them at one. The finding is buried in the Ahrefs bot breakdown: "GPTBot is top and Claude-Code is second, ahead of every AI search and assistant bot." Claude Code out-fetched every AI retrieval bot, every assistant, and every training crawler.

The qualifier does real work. Ahrefs' reading is that "agents fetch llms.txt when directed, not speculatively" — no agent discovers your file by wandering in. A developer pastes a URL, or your docs mention the path. So the practical move is to link it from your HTML and reference it in your docs, rather than publishing and waiting.

If your users run coding agents against your documentation, the file has a real audience. If they do not, the 1.1% retrieval-bot number is your realistic expectation. And if you are on the reading side of that workflow, agent-written markdown has its own problems — I wrote about reading Claude Code plans before approving them separately.

Writing One That Is Worth Publishing

The spec is small enough to write by hand in ten minutes, and hand-writing is how you avoid the failure modes in the 45% malformed figure.

• Include the blockquote summary. It is what auto-generators most often skip, and it gives a model context for everything under it.
• Curate, do not dump. Humphrey's description of plugin output is exact: "dumping every page, post, and category into one list technically produces a file, but it is not the curated map the format was designed to be."
• Make robots.txt and llms.txt agree. 57 of the 737 sites in Humphrey's scan, about 8%, publish an llms.txt while blocking AI crawlers in robots.txt. His takeaway: "whatever you decide about AI crawlers, decide it once and make robots.txt and llms.txt agree."
• Serve it as real markdown or plain text. Thirty percent of apparent adopters were serving an HTML soft 404, which no amount of good content inside fixes.
• Keep it boring. Ahrefs' security guidance is to treat the file like code: version-control it, restrict who can edit it, and keep it to plain links and descriptions with nothing instruction-shaped. Agents are built to trust this file, so a stale or tampered one misleads every agent that reads it. The largest research crawler hitting these files identifies as prompt-injection-survey/1.0. No exploit has been reported in the wild; people are clearly looking.

If you would rather generate one: Mintlify hosts an llms.txt automatically for docs sites, Yoast, All in One SEO and Rank Math ship WordPress generators, and Answer.AI publishes the reference llms_txt2ctx CLI. Whatever produces it, read the output before shipping it.

How Do You Check an llms.txt File?

Open it as markdown, because it is markdown. Served as text/plain, an llms.txt renders in a browser as an unstyled wall of text: no headings, no clickable links, no section breaks. Survivable for Svelte's 1.7 KB file. For Stripe's 93 KB across 26 sections, that is not a review, it is scrolling.

The workflow I use takes about two minutes:

1. Fetch it: curl -s https://example.com/llms.txt -o llms.txt
2. Rename it to llms.md. The extension is the only friction, since viewers key off it.
3. Open it rendered. Heading structure, link text and section boundaries all become visible at once, which is exactly the view you need to answer "is this actually curated, or did something dump my sitemap into it?"
4. Verify every link resolves, then check robots.txt does not block the bots you just invited.

On a Mac, step 3 can be a Finder preview: select the file, press Space, and QuickLook renders it in place. MacMD Viewer also reloads the open document when another process writes to it, so you can keep the rendered file on one side and edit on the other, watching the structure change as you cut links.

What My Own llms.txt Looks Like

I have spent this article asking for receipts, so here are mine. macmdviewer.com/llms.txt is 14,243 bytes served as text/plain, with one H1, twelve H2 sections and ninety link-list items, all in - [name](url): notes form. It ends with a genuine ## Optional section holding the privacy policy and terms, which is exactly what that reserved section is for. I also follow the spec's companion proposal, rarer than llms.txt itself: clean markdown versions of pages at the same URL with .md appended, so macmdviewer.com/pricing.md returns 691 bytes of text/markdown.

And one thing I got wrong, since a guide that only shows its author's wins is not much of a guide: my file has two blockquotes. The spec describes a single summary blockquote; my second points at llms-full.txt. Small drift, nobody has complained, and a fair illustration of how easily a hand-maintained file wanders from a spec you read once.

Worth saying plainly: I published my GEO playbook in April 2026, and it recommends adding an llms.txt with more confidence than the 2026 data now supports. The file is still in my setup. The reason I keep it has changed.

Frequently Asked Questions

What is an llms.txt file?

An llms.txt file is a markdown file served at the root of a domain that gives language models a curated map of the site. It was proposed by Jeremy Howard of Answer.AI on 3 September 2024, and the spec at llmstxt.org describes it as a way "to provide information to help LLMs use a website at inference time." It is not a crawler directive: it allows nothing, blocks nothing, and controls nothing.

Does llms.txt help SEO or AI rankings?

No, on the evidence available as of July 2026. Google Search Central states that machine-readable AI text files "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them." Ahrefs' June 2026 study of 137,210 domains found 97% of llms.txt files received zero requests in May 2026, and only 1.1% of the requests that did happen came from AI retrieval bots.

Is llms-full.txt part of the llms.txt spec?

No. The llmstxt.org spec never defines llms-full.txt. The companion files it does describe are llms-ctx.txt and llms-ctx-full.txt, generated by the llms_txt2ctx tool. The name llms-full.txt is a de facto convention popularised by documentation platforms, chiefly Mintlify, meaning an entire docs site concatenated into one file.

How many websites have an llms.txt file?

It depends entirely on the population you measure. Ahrefs found 28% of the 137,210 domains in its Web Analytics data publish one, and stated that figure should be treated as an upper bound because its customers skew technical. Chris Humphrey's June 2026 scan of the top 10,000 Majestic Million domains found 7.4%, or 737 sites, publishing a real markdown file.

What is the correct llms.txt format?

An H1 with the project name, then optionally a blockquote summary, then optional prose containing no headings, then H2 sections holding markdown link lists in the form of a name, a URL, and optional notes after a colon. The H1 is the only required element. A section titled Optional has reserved meaning: its URLs can be skipped when a shorter context is needed.

Do ChatGPT, Claude, or Google read llms.txt?

None of them document consuming it. As of 26 July 2026, the crawler documentation from Google, OpenAI and Anthropic does not mention llms.txt at all, and robots.txt is the stated control mechanism in every case. Google names the file once, in its AI optimization guide, and only to say Search ignores it: "It's completely fine if you decide to create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files." Coding agents are the exception in practice: Ahrefs found Claude Code fetched llms.txt files more often than any AI retrieval bot, assistant or training crawler.

How do I check my llms.txt file is valid?

Fetch it, save it with a .md extension, and open it in a markdown viewer so the heading structure and links render. Then check it against the spec: exactly one H1, a blockquote summary, H2 sections whose items are all links, and an Optional section only if you mean the skippable semantics. Finally, confirm every link resolves and that robots.txt does not block the bots you invited.

Should You Publish One?

If your users point coding agents at your documentation, yes. It costs an hour, it is the one use case with measured demand, and a curated index is a genuinely better starting point for an agent than your sitemap.

If you are publishing it to appear in ChatGPT or AI Overviews, the honest answer as of July 2026 is that you are decorating. Google says it ignores the file, no provider documents reading it, and 97% of the files that exist got no requests at all last May. That may change. Standards have gone from ignored to expected before, and an hour spent now is a cheap option on it.

What I would not do is let the file rot. An llms.txt that lists pages you deleted is worse than none, because the one audience that does read it takes it at face value. Fetch yours, open it rendered, and see whether it still describes the site you have.