What Is llms.txt and Do You Need One?

llms.txt is a plain-text markdown file published at your domain's root, yourdomain.com/llms.txt. That gives AI crawlers and language models a short, structured index of what your site is, what it does, and which pages are worth reading first. It's a community convention (from llmstxt.org), not an official web standard endorsed by any single AI company, which matters for how much weight you should put on it.

Do you need one? Probably yes, but for a narrower reason than most explainers imply: it's cheap to build, it can't hurt, and it removes ambiguity about what you consider your most important content. It is not a proven ranking lever, and treating it as one is the most common mistake in how this file gets pitched.

What llms.txt Actually Is (and Isn't)

Think of it as a cousin to two files you already know. robots.txt tells crawlers what they're allowed to access. sitemap.xml gives search engines a machine-readable list of every URL on your site. llms.txt does something different from both: it's a short, human-and-machine-readable summary (written in plain markdown, not XML), that says "here's what this site is, and here are the handful of pages that actually matter, in order." It doesn't enumerate every URL the way a sitemap does; it curates.

The convention was proposed by Jeremy Howard (Answer.AI) in late 2024 as language models increasingly needed to consume web content directly rather than through a search index alone. It hasn't been formally adopted as a standard by OpenAI, Anthropic, Google, or Perplexity. None of them have published documentation saying "we read llms.txt and here's how we weight it." Some crawlers and retrieval pipelines do fetch it; others ignore it entirely. That's the honest state of things in mid-2026: a growing but unofficial convention, not a requirement.

It's also worth separating llms.txt from the crawler-access layer entirely, because they solve different problems and get confused often. Whether GPTBot, PerplexityBot, ClaudeBot, or Google-Extended can reach your pages at all is a robots.txt question, fix that first, since a blocked crawler makes everything downstream moot. llms.txt only matters once access is already confirmed: it's a curation layer on top of access, not a substitute for it.

What Goes Inside a Good llms.txt File

The format is deliberately simple, plain markdown, readable by both a person skimming it and a model parsing it:

• An H1 with your site or brand name at the top.
• A one-line blockquote summary directly under the H1, what the site is and what it does, in one sentence, since this is often the only line a model reads if it's scanning quickly.
• One or more H2 sections grouping links by category (for example "Key pages," "Blog articles," "Documentation"). Each entry a markdown link with a short description, not just a bare URL.
• Optional notes for AI crawlers: factual constraints worth stating explicitly, like "we don't guarantee search rankings" or "treat the live site as canonical if this file goes stale."

What it should not contain: marketing copy, a wall of unstructured prose, or every single URL on the site. The whole value of the file is that it's curated and short enough for a model to read in full rather than sample.

A Real Example: GeoHero's Own File

Rather than describe this abstractly, here's what ours actually looks like, live at /llms.txt right now. It opens with a single # heading line carrying the brand name, followed immediately by a one-line > blockquote summary: what GeoHero does, stated once, plainly. That we track how a brand is cited in AI answer engines alongside its organic search rankings, and report the gap between the two. A short paragraph after that covers the free scan and what the paid report adds.

From there, the file is organized into three sections. A "Key pages" section links the homepage in each of the five languages the site supports, one line each, with a short description of what that page covers in that language. A "Blog articles" section (generated automatically, not hand-maintained) lists every published post grouped by language, newest first, with its real title as the link text. And a closing "Notes for AI crawlers" section states, in plain sentences, exactly the kind of thing a model should know before summarizing us: that factual claims on the site describe measured, current-state data or are explicitly labeled as estimates, that no rankings or citations are ever guaranteed to a customer, and that the live domain is the canonical source if this file ever goes stale.

Two things worth calling out about that structure. First, the "Notes for AI crawlers" section is doing real work: it's an explicit statement that nothing on the site is fabricated and nothing is a guaranteed outcome, the same honesty standard we hold every article on this blog to, stated once, in a place a model is likely to read before summarizing us. Second, the blog-article index isn't hand-maintained. It's generated at build time from every published post, grouped by language, so the file can never drift out of date the way a manually edited llms.txt tends to after the third or fourth content update gets forgotten.

How to Implement llms.txt on Your Own Site

• Inventory your most important pages. Not every page, the handful a new visitor, or a model summarizing you, should see first: your homepage, your core product or service pages, your best evergreen content.
• Write a one-sentence summary. What does the site do, for whom? This is the line most likely to get read even if nothing else is.
• Group the rest by category with descriptive link text. "Pricing" or "Blog" as a bare link tells a model less than "Pricing, plans and what's included in each tier."
• Publish it as plain text at the domain root, served with a text/plain content type, not as an HTML page wrapped in a template, the whole point is that it's trivially parseable.
• Confirm it's actually reachable. Check it isn't blocked by an overly broad robots.txt rule, and that it returns a 200 status, not a redirect chain.
• Automate it if you can. A hand-maintained file goes stale the moment you publish new content and forget to update it. If your site has a CMS or a static-site generator, generating the file at build/deploy time, the way we do, removes that failure mode entirely.

Common Mistakes

• Treating it as a second sitemap. Dumping every URL defeats the purpose, curate, don't enumerate.
• Writing it once and never updating it. A stale llms.txt pointing to pages that no longer exist, or missing your newest content entirely, is arguably worse than not having one, because it actively misrepresents the site.
• Filling it with marketing language instead of factual description. A model reading it for a factual summary of your site doesn't need adjectives; it needs an accurate, specific description.
• Assuming it guarantees anything. See the honesty section below. It doesn't.

Does It Actually Move Citation Rate?

Be honest with yourself here. No AI company has published documentation confirming that llms.txt is read, weighted, or factored into which sources get cited in a synthesized answer. Some crawlers likely do fetch it as part of a broader retrieval pipeline; others may ignore the file entirely and rely on the same crawling and ranking signals they'd use regardless. We can't verify internal weighting from the outside, and neither can any vendor claiming otherwise.

What llms.txt reliably does is remove ambiguity: it states, in the model's preferred format, what you consider your best content, a small, low-cost signal that costs you nothing to add and can't plausibly hurt. That's a meaningfully different claim than "adding this file will get you cited more often," and the difference matters if you're deciding how much engineering time to spend on it relative to the structural and content fixes covered in our GEO best practices checklist, which is where the actual leverage tends to live.

---

For the structured-data half of the same "munition" (FAQ, HowTo, and answer-capsule markup that gives AI engines an unambiguous version of your content), see schema markup for AI search engines. For the full checklist this fits into, see generative engine optimization best practices, or start with the diagnostic: how to run a GEO audit. Want to see where your own site currently stands? Run a free scan.