If you’ve seen blog posts in the last twelve months pitching llms.txt as “the new robots.txt for AI”, they’re wrong on the substance. llms.txt is something else. Worth getting right because the alternative (your site getting summarized poorly by AI search engines that can’t parse your nav) is real.

This piece covers what llms.txt actually is, what it isn’t, the parts of the spec that matter, real examples you can copy, and which AI clients use it as of July 2026 (spoiler: fewer than the hype suggests).

What llms.txt is

llms.txt is a markdown file at the root of your domain that summarizes your site for large language models. It’s a content document, not a permission directive. It points an LLM at the canonical pages it should read if it wants to understand what you do.

Proposed by Jeremy Howard (Answer.AI) in September 2024 and tracked at llmstxt.org, it solves a specific problem: when an LLM grounds an answer in your site, it has a limited context window. It cannot read your entire sitemap, fetch every linked page, parse JavaScript-rendered nav, and produce a coherent answer in the few seconds the user is waiting. So it picks the wrong pages, misses your pricing, summarizes outdated marketing copy, or just hallucinates.

llms.txt is your chance to say: here is the actual canonical summary, here is the one-line description, here are the URLs that matter for the questions humans will ask.

The file lives at https://yourdomain.com/llms.txt. Same convention as robots.txt and sitemap.xml.

What llms.txt isn’t

Three myths that show up in 90% of “llms.txt explained” blog posts:

Myth 1: it’s robots.txt for AI. No. robots.txt is a crawl-control directive: it tells bots which paths they may or may not fetch. llms.txt doesn’t control access at all. It’s a content artifact, not a policy artifact. The two are complements, not alternatives.

Myth 2: it’s mandatory and AI engines fail to find your site without it. Also no. AI search engines (Perplexity, ChatGPT Search, Google AI Overviews, Claude’s web tool) crawl HTML and follow the same signals as Google: heading hierarchy, schema.org, sitemap.xml, internal linking. A site without llms.txt ranks fine. And the honest flip side: there’s no measured evidence that a site with one gets cited more, because the major crawlers mostly don’t fetch the file yet (details in the adoption section below). The case for it is that it’s nearly free, can’t hurt, and doubles as a clean site summary for any agent that does look.

Myth 3: a standards body is enforcing it. Also no. As of mid-2026 the spec lives at llmstxt.org as a proposal. Adoption is voluntary. Some major sites ship one (Anthropic on their docs domain, Vercel, FastHTML, Cursor); most don’t.

What the spec actually says

The format is intentionally minimal. A valid llms.txt is:

# Project Name

> One-line description of what you do.

Optional paragraph(s) of detail.

## Section heading

- [Page title](url): one-line description
- [Another page](url): another description

## Optional

- Less important links here.

That’s it. Markdown headers (#, ##), bullet links with descriptions, optional prose. No YAML frontmatter, no JSON, no embedded code beyond markdown.

Two sub-conventions worth knowing:

  • ## Optional is a special heading: LLMs are advised to drop these sections first if context is tight. Use it for tangential content.
  • A companion file llms-full.txt can ship the full content of each linked page expanded inline. Useful if your important pages are small enough to fit, since the LLM doesn’t need to fetch them. Most sites don’t need both; start with llms.txt only.

A real example: ours

Here’s the mcp-analytics.com/llms.txt, trimmed:

# mcp-analytics

> Web analytics you query through Claude or any MCP client. No dashboard,
> no charts, just answers. Free up to 100,000 hits/month, EU-hosted in
> Germany, cookieless and cookie-banner-free by default.

mcp-analytics turns web analytics into a conversation. The primary
interface is the Model Context Protocol (MCP). Instead of opening a
dashboard, users connect their account to Claude Desktop, ChatGPT custom
connectors, Cursor, or any MCP client and ask questions like "top pages
last 7 days" in natural language. The MCP server exposes 23 read/write
tools covering overview metrics, top pages, referrers, UTM sources,
custom events, AI-crawler classification, engagement, and account
management. Tracking is cookieless by default and all data stays in
the EU.

## Product surface

- [Landing page](https://mcp-analytics.com/): product positioning,
  pricing tiers, signup form.
- [Docs](https://mcp-analytics.com/docs): human-readable setup guide.
  Connect via OAuth from any MCP client and start asking questions.

## MCP tool catalog

- [All MCP tools](https://mcp-analytics.com/mcp/tools): index of every
  callable function with arguments and example prompts.

## Content

- [Blog](https://mcp-analytics.com/blog): tutorials and essays on MCP,
  AI workflows, and web analytics. German blog at
  [/de/blog](https://mcp-analytics.com/de/blog).
- [Comparisons](https://mcp-analytics.com/vs): honest side-by-sides
  versus Plausible, Fathom, Pirsch, Umami, PostHog, Google Analytics.
  German at [/de/vs](https://mcp-analytics.com/de/vs).

## Optional context

- Architecture: Rails 8 plus a Go ingest service plus ClickHouse,
  deployed via Kamal on a single EU VPS.

Three deliberate choices:

  1. The blockquote in the second line gives Claude/ChatGPT a copy-pasteable one-liner. If an assistant quotes anything back about mcp-analytics, this is the sentence we want it to be. Treat it like meta-description squared.
  2. Section headings reflect user intent. “Product surface”, “MCP tool catalog”, “Content” are how a user would categorize what to read, not how we categorize internally.
  3. Architecture goes under the optional section. Most users don’t need to know we run on a small EU VPS. Putting it there means: if the LLM has 5 tokens left, drop this first.

Other notable examples worth copying from

Site What’s good about it
platform.claude.com/docs/llms.txt Anthropic’s own, on the docs domain (formerly docs.anthropic.com; the bare anthropic.com root doesn’t serve one). Strong section labels, clean separation
vercel.com/llms.txt Heavy use of llms-full.txt for docs that the LLM can answer without follow-up fetches
fastht.ml/docs/llms.txt Reference implementation from the spec author
docs.stripe.com/llms.txt Massive docs corpus, granular structure. Useful pattern for API-heavy sites
cursor.com/llms.txt Two-tier structure: product features plus docs, clear which to read for which question

Worth opening all of these in tabs. The patterns repeat: short blockquote summary, scannable section headers, optional section at the bottom.

Which AI clients actually read it (July 2026)

This is the question that decides whether you should bother, and the honest answer is: adoption is thin. Independent crawl-log analyses agree that the big AI crawlers mostly don’t request the file. Client by client:

  • Google AI Overviews: Google has said outright they don’t use llms.txt. Existing crawl plus Knowledge Graph instead.
  • ChatGPT search: published crawl-log analyses conclude OAI-SearchBot generally does not fetch it.
  • Perplexity: PerplexityBot essentially never requests it in the crawl logs people have published. (An earlier version of this post claimed Perplexity had blogged about using llms.txt as a ranking signal. We couldn’t verify any source for that, so we’ve removed the claim.)
  • Claude: llms.txt fetches show up rarely in crawl logs, and there’s no documented support.
  • Cursor: you can paste an llms.txt URL into @Docs as a custom doc, and the file’s shape makes it a good fit for that. But that’s you adding it manually; Cursor doesn’t fetch it on its own, and it’s not documented as an official llms.txt integration.
  • Phind, You.com, DuckDuckGo AI: no signal either way. Safe assumption: standard HTML/sitemap parsing.

Practical takeaway: don’t ship llms.txt expecting a measurable citation boost; nobody can honestly promise one today. Ship it because it costs one markdown file, can’t hurt, doubles as a clean site summary any agent (or human) can read, and is already in place if a major client starts fetching it.

How to write one for your own site

Forty minutes from zero. Steps:

1. List the 5 questions your users will ask an LLM about you

If you sell a SaaS, those are typically:

  • “What does X do?”
  • “How much does X cost?”
  • “Is X better than Y?”
  • “How do I set X up?”
  • “Does X support Z (a specific feature)?”

For each question, identify the single canonical page that answers it best.

2. Write the blockquote sentence first

It’s the sentence you most want repeated when an LLM describes you. Test it: copy it into a new chat, paste with “explain what this product is in your own words”, check that the LLM’s paraphrase matches what you want.

3. Group the canonical pages into 3-5 sections

Headings matter for context. Good labels: “Product”, “Pricing”, “Docs”, “Comparisons”, “Changelog”. Bad labels (too generic for an LLM to use as routing): “Stuff”, “Links”, “Important pages”.

4. Each bullet: [Title](url): one-line description

Don’t restate the title in the description. The pattern that works:

- [Pricing](https://yoursite.com/pricing): free up to 100k events, paid €19/mo, no annual.
- [Comparisons](https://yoursite.com/vs): honest side-by-sides vs Plausible, Fathom, GA.

NOT:

- [Pricing](https://yoursite.com/pricing): The pricing page.
- [Comparisons page](https://yoursite.com/vs): A page that compares us to other tools.

5. Drop anything tangential into ## Optional

Your architecture page, your founder’s blog, your changelog. Useful but skippable.

6. Ship it to public/llms.txt

For most stacks: drop the file in the static directory and you’re done. Rails: public/llms.txt. Next.js: public/llms.txt. Astro: public/llms.txt. Hugo: static/llms.txt. Jekyll: the project root (root-level files are copied into the built site as-is).

Verify with curl https://yourdomain.com/llms.txt. Make sure Content-Type: text/markdown (most static servers will set this; some default to text/plain and that’s fine too, LLMs handle both).

7. Reference it from robots.txt

Not required, but a few crawlers pick it up faster if you do:

Sitemap: https://yourdomain.com/sitemap.xml

# Pointer to llms.txt for AI clients that look for it.
# Not part of the robots.txt spec; treat as a hint.

Adding llms.txt to your sitemap won’t hurt either.

llms.txt and the bigger AI-search picture

llms.txt is one of three on-page surfaces that can affect whether an LLM cites your site:

  1. Standard SEO signals. Title, meta description, heading hierarchy, internal linking, schema.org JSON-LD. These still dominate. AI search is built on top of regular search, not next to it.
  2. llms.txt. The hub document we just covered. Today more forward-looking than measurably effective.
  3. robots.txt AI-crawler allowance. If you Disallow: / for GPTBot, ClaudeBot, or PerplexityBot, you’re invisible in those engines. Common mistake: people who installed a “block AI crawlers” plugin in 2024 to protest training, forgot to undo it when they later wanted to be in ChatGPT’s answer set.

If you only do one of the three, do #3 (allow AI crawlers). If you do two, do #3 plus #2. Combined cost: half an hour, lasts forever.

Common mistakes that look right but break things

  • Putting llms.txt behind auth or geo-block. Crawlers fetch it without sessions. Make sure it returns 200 from a cold-cache anonymous fetch.
  • Linking to pages that 404 or are noindex. Test every URL. The fastest way: curl -I each link, check 200.
  • Using llms-full.txt to dump 100k tokens of marketing copy. LLMs have context budgets. A 100k-token llms-full.txt gets truncated. Keep it lean.
  • Writing it in the third person about yourself. “Acme Corp is a leading provider of cloud-based…” reads like a press release and LLMs paraphrase it that way. First person (“we”) or second person (“you”) is better.
  • Forgetting to update it when pages move. When you rename /features to /product, update llms.txt in the same commit. Build it into your release checklist.

Where this is going

llms.txt is not even two years old as a proposal (September 2024). Three things that could happen in the next 12-18 months:

  • AI clients adopt it explicitly. None of the big ones formally have yet, and we won’t pretend to know if they will. If one does, sites that already ship the file are ahead by exactly one commit.
  • The spec adds optional metadata. Versioning (which version of your product the doc reflects), last-modified timestamps, language tags. Right now the format is pure markdown, no header for “this file is for the v2.3 docs”. Expect that to change.
  • Tooling. Static-site generators will ship llms.txt plugins. Vercel/Netlify will likely auto-generate one from sitemap plus meta-descriptions.

For now: write yours manually, keep it ~50 lines, revisit every quarter.

If you’re building anything LLM-adjacent

Two complementary surfaces are worth setting up alongside llms.txt:

  1. An MCP server, if you have data that LLMs should query live (not just summarize). We wrote a practical Claude MCP setup guide that covers building and connecting one.
  2. robots.txt allow-list for the big AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Applebot-Extended). Without this, llms.txt doesn’t matter because nobody reads it.

For mcp-analytics (the product we’re building) both are in production. The MCP server is the product. llms.txt and the AI-friendly robots.txt are how new users find out the product exists.

If web analytics that lives entirely inside Claude/ChatGPT sounds interesting: sign up free, no credit card, 100k hits/month included. We’ll let you know when the first user gets here through an AI search citation.