← Journal

Guides

llms.txt explained: what it is, how to write one, and whether Google uses it

By Thomas Lovas Lokoy · Published 23 October 2026 · 6 min read

A vintage computer terminal showing green plain text on a dark screen
Photo: Zenith Z-19 Terminal by ajmexico · CC BY 2.0

An llms.txt file is a short Markdown file at the root of a website, at /llms.txt, that gives AI assistants and agents a curated map of the site: what it is, and links to the pages that matter, ideally in clean Markdown. It is a community proposal, not an official web standard, and Google says you do not need it, or any other AI text file, to appear in its AI features. It is cheap to add and may help tools that read it; it will not change how Google ranks you. Here is what the proposal actually specifies, how to write one, and how to decide whether it is worth your time.

Where llms.txt came from

The proposal was written by Jeremy Howard and published on llmstxt.org on 3 September 2024; the current version of the page, v2, is dated 10 August 2026. Its stated aim is to "standardise on using an /llms.txt file to provide information to help agents use a website".

The reasoning is practical. A web page built for people carries navigation, scripts, cookie banners and layout that an AI system has to strip away to find the content. A short, curated Markdown file, pointing to Markdown versions of the important pages, is easier for a language model to use within its limited context.

What goes in the file

The format is plain Markdown so that both models and ordinary parsers can read it. The proposal lays out the sections in a fixed order:

| Section | Required? | What it holds | | --- | --- | --- | | H1 heading | Yes, the only required part | The name of the project or site | | Blockquote | Optional | A short summary of the project | | Detail paragraphs or lists | Optional | Any further context, with no headings | | H2 sections with file lists | Optional | Markdown links, written as name, each optionally followed by a colon and a note | | An H2 titled "Optional" | Optional, by convention | Secondary links an agent can skip when it needs less context |

The file normally sits at the site root. The v2 proposal also allows it at a subpath, with each file covering the URLs below it, and says agents should use the most specific file that applies.

The Markdown versions of pages

The second half of the proposal matters as much as the file itself. It asks sites to "provide a clean markdown version of those pages at the same URL as the original page", and describes two link relations for finding them:

  • rel="alternate" type="text/markdown" on a page points to its Markdown version.
  • rel="describedby" points to the llms.txt file that covers the page.

These can be HTML link elements or an HTTP Link header. If you publish an llms.txt but your linked pages are full HTML, you have done the easy half.

A worked example: this site's own file

AI Orbit publishes one at aiorbit.site/llms.txt, so you can see the format in use. It has an H1 with the product name, a blockquote summarising what the product does, a short paragraph on how billing works, an H2 listing the plans, an H2 listing the main pages, and H2 sections listing every journal post with its description. The plan figures in it are generated from the same pricing code the site sells from, so the file cannot quote a price the product does not charge.

A minimal file for a small business blog might look like this:

# Example Bakery

> A family bakery in Bergen. We sell sourdough and pastries in our shop and publish baking guides.

## Guides
- [Sourdough starter guide](https://example.com/guides/starter.md): how to make and feed a starter
- [Opening hours and location](https://example.com/visit.md): where to find us

## Optional
- [About us](https://example.com/about.md)

How it differs from robots.txt and a sitemap

The three files are often mentioned together, but they do different jobs:

| File | Who it is for | What it does | | --- | --- | --- | | robots.txt | Crawlers | Says which paths a crawler may or may not fetch | | sitemap.xml | Search engines | Lists the URLs you want discovered, with optional dates | | llms.txt | AI assistants and agents | Gives a short, curated, human-readable map of the site and its key pages |

Only robots.txt is about permission, and only a sitemap is a complete list. An llms.txt is deliberately selective: the proposal frames it as an overview that points to more detailed Markdown, not as an index of every URL. If a page matters enough to be in your llms.txt, it should already be in your sitemap and crawlable under your robots.txt.

Common mistakes

  1. Listing every URL. That is a sitemap's job. Keep the file short enough to read in a minute.
  2. Linking to pages that redirect or 404. Check every link when you publish and whenever you restructure the site.
  3. Writing marketing copy. The blockquote should say plainly what the site is; an assistant summarising it will repeat what you wrote.
  4. Letting it go stale. A file that lists last year's prices or a discontinued product is worse than no file.

What llms.txt does not do

Three claims about llms.txt circulate that the sources do not support:

  1. "It helps you rank in Google." Google's guidance on AI features says: "You don't need to create new machine readable files, AI text files, or markup to appear in these features." The requirements for AI Overviews and AI Mode are the same as for a normal snippet: the page must be indexed and eligible to be shown with a snippet. Our guide to appearing in AI Overviews covers what Google does recommend.
  2. "It controls AI training." Nothing in the proposal is a permission or a block. It is a map, not a rule. Crawling is controlled by robots.txt, and Google documents a separate control, Google-Extended, for some of its AI uses.
  3. "It is a standard." It is a proposal published by its author. Whether a particular assistant reads it is up to that assistant's maker.

Should you add one?

A short decision guide:

  • Add one if your site has documentation, guides or product information that people ask AI assistants about, and you can keep it current. It takes under an hour for a small site.
  • Generate it from your data if you can. A file written by hand drifts as pages change. Building it from the same source as your sitemap or content list keeps it accurate, which is how this site's file is built.
  • Skip it, for now, if you have not yet done the basics: indexable pages, internal links and clear answers. Those affect every search and AI system; an llms.txt affects only the tools that choose to read it.
  • Never use it to say things your pages do not. If the file describes a price, feature or claim, the linked page should say the same. Treat it like any other public copy, and check it like an article.

A five-step setup

  1. Write the H1 and a one-sentence blockquote describing the site.
  2. List your ten most useful pages under one or two H2 headings, each with a short note.
  3. Put secondary pages under an H2 called Optional.
  4. Publish it at /llms.txt as plain text, and link to Markdown versions of pages where you have them.
  5. Add a reminder to review it whenever you change prices, plans or your most important pages; our content calendar guide shows one way to schedule that.

Where AI Orbit fits

Every article AI Orbit writes is delivered as clean Markdown, which is the format llms.txt points to, so the Markdown version of an article is the draft you already have. You still decide what to publish and where. Plans and credits are on the pricing page.

Sources