Guide

llms.txt done right

llms.txt is a small markdown file at your web root that tells an AI agent what your site is and where the pages and machine endpoints that matter actually live. Done well it saves an agent from guessing; done badly it is noise an agent learns to ignore. Here is the difference.

What it is, precisely

A plain-text, markdown-structured file served at https://your-domain/llms.txt with a text/plain (or text/markdown) content type. It opens with an #H1 naming the project, a short paragraph of what you do, then curated sections of links — each link annotated with why an agent would follow it. That’s it. It is a map written for a reader who arrives by probing, not by browsing.

The key word is curated. A sitemap lists every URL; llms.txt lists the handful that orient an agent: what you are, how to reach your API or MCP endpoint, where the canonical docs are. If everything is important, nothing is.

The format

# Acme Gateway

> One-line positioning an agent can quote back: what this is and who it's for.

A short paragraph — two or three sentences — describing what the project does
in plain language. No adjectives you can't back up.

## Core
- [Home](https://acme.example/): positioning and feature overview
- [Docs](https://acme.example/docs): how it works, in depth
- [API / MCP endpoint](https://mcp.acme.example/mcp): live endpoint agents connect to

## For agents
- [AGENTS.md](https://github.com/acme/gateway/blob/main/AGENTS.md): verifiable claims
- Repository: https://github.com/acme/gateway (MIT)

## Notes
- Status: beta. Breaking changes tracked in the changelog.

Headings group links; each bullet is [label](url): one line on why it matters. An agent can read this top to bottom and know exactly where to go next.

Anti-patterns

Do

  • +Write it by hand and keep it short — a page of links, not a database dump.
  • +Annotate every link with the reason an agent would follow it.
  • +Point to your machine surfaces: API, MCP endpoint, AGENTS.md, feeds.
  • +Serve it as text/plain with a 200, and keep counts/claims current.

Don’t

  • −Paste your whole sitemap — hundreds of unannotated URLs is noise.
  • −Write marketing prose. "Revolutionary AI-powered synergy" tells an agent nothing.
  • −Let a catch-all route serve HTML (your 404 or SPA) at /llms.txt.
  • −Hardcode stale numbers ("50+ tools") that drift from reality.

The failure that hurts most is the silent one: a framework catch-all returns your app shell at /llms.txtwith status 200. Nothing errors, so you never notice — but every agent gets HTML where it expected markdown. This is why the last step is always to fetch the bytes, not to trust the deploy.

An annotated real example

The llms.txt of cortex-gateway.dev follows the shape above: an H1 and a one-line positioning, a paragraph of what it is, then sections that send an agent straight to the live MCP endpoint, the repository, the docker image, and an AGENTS.md full of claims it can verify. It names its live demo endpoint explicitly (https://mcp.cortex-gateway.dev/mcp) rather than making an agent guess the path.

Notice what it does not do: no feature marketing, no list of every doc page, no superlatives. Every line is either a fact or a link an agent will actually use.

Verify it

Fetch the bytes and read the first line. HTML here is a failure even at status 200:

curl -s https://your-domain/llms.txt | head -5
# PASS: starts with "# Your Project"
# FAIL: starts with "<!doctype html>" — a route is serving your app shell

Then run the whole domain through the agent-readiness checker: it flags an llms.txt that returns HTML, is missing, or is nearly empty — and shows you the exact curl for each verdict.

FAQ

Where does llms.txt go?

At the web root: https://example.com/llms.txt. It is a root-level convention like robots.txt, not something under /.well-known/. One file per origin.

Does llms.txt replace robots.txt or a sitemap?

No. robots.txt tells crawlers what they may fetch; a sitemap enumerates URLs for indexing; llms.txt is a curated, human-written map that tells an agent what your site is and where the load-bearing pages and machine endpoints are. They serve different readers and should all coexist.

Do agents actually read it?

Increasingly, yes — assistants and crawlers probe well-known paths before scraping HTML. But treat it as a bonus channel, not a guarantee: everything in llms.txt must also be true and reachable on the real site, because an agent will verify before it trusts.

llms.txt vs llms-full.txt?

llms.txt is the short index (a map with links). Some sites also publish llms-full.txt with the full expanded text inline for agents that want everything in one fetch. Start with llms.txt; add the full variant only if you have the content to justify it.

What is the single most common mistake?

Serving your SPA or 404 page at /llms.txt. A catch-all route returns HTML with a 200 status, so an agent gets your homepage dressed as a machine file. Always curl it and confirm the body starts with your markdown, not <!doctype html>.
Verify any of this on a real domain with the agent-readiness checker— every result line ships the curl that reproduces it. See the pattern shipped end-to-end in cortex-gateway.