1. What llms.txt Is

llms.txt is a convention: put a Markdown file at the root of your site, and use plain language to tell large language models and AI agents what the site is, which content is worth reading, and what each piece covers.

It was proposed by Jeremy Howard (Answer.AI) in September 2024, and the entire specification fits on one page.

Put simply, robots.txt governs “whether you may crawl”; llms.txt governs “how to make sense of what you crawled” — the former is permission, the latter a reading guide.

• Specification: https://llmstxt.org/
• Reference implementation: https://github.com/AnswerDotAI/llms-txt

A minimal, usable llms.txt looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

# Shaowen Chen's Website

> A personal engineering blog about AI, cloud native, and infrastructure, with 700+ hands-on notes.

The site is primarily in Chinese, with English translations for much of it. It covers
practical work in Kubernetes, Ceph, RDMA, GPU operations, and AI agents, and every
post pairs its conclusions with reproducible commands.

## Core Content

- [Ceph Architecture and Operations](https://www.chenshaowen.com/en/blog/container-deploy-ceph-architecture-operations-and-testing.html): cluster architecture, routine inspection, monitoring, and upgrades
- [RDMA Operations](https://www.chenshaowen.com/en/blog/rdma-ops.html): configuring and testing RoCE, Soft RoCE, and InfiniBand

## Optional

- [About](https://www.chenshaowen.com/en/about.html): how to get in touch

Note that the whole file is ordinary Markdown, written for humans and models alike: no XML, no JSON Schema, no config file. The entire complexity of the convention is “say what you mean clearly.”

2. Why You Need It

2.1 An HTML Page Is Not Model-Friendly

Feed a page to a model and only a small fraction of it is useful body text; the rest is navigation, sidebars, ad slots, cookie banners, inline scripts, and styles. The model has to run a pass of “extract the body text” first, which burns tokens and often gets it wrong.

The harder problem is discovery. When a site has hundreds of posts, a model has no good way to know “which few are central, which are already outdated.” It can only walk the sitemap, rely on search hits, or guess at URL patterns.

The idea behind llms.txt is: rather than making the model guess, tell it outright.

2.2 Comparison: With and Without llms.txt

Suppose a user asks an AI, “How do I troubleshoot a Ceph problem?”

Without llms.txt:

Fetch the homepage → it lists only the latest 5 posts → 140+ pages to page through
     → switch to sitemap.xml → thousands of URLs, mostly tag and pagination pages, no way in
     → match 3 by keyword → fetch each HTML page, extract the body, strip template noise
     → 1 of the 3 is a years-old post whose commands no longer work
     → answer given (a large share of tokens spent on templates and stale posts)

With llms.txt:

Request /llms.txt → get the site's positioning + the curated list of core posts + a one-line summary of each
     → go straight to "Ceph Architecture and Operations"
     → fetch the body → answer given

2.3 It Is Not SEO

Worth stating up front: llms.txt does not improve search ranking, and Google has said explicitly that it does not use it as a ranking signal.

It serves a different path — how AI search, AI browsers, and coding agents draw on your content when answering questions. That area is usually called GEO (Generative Engine Optimization). SEO feeds pages to crawlers; GEO feeds understanding to models.

2.4 A Side Benefit: A Site’s Self-Description

llms.txt has a value that is easy to overlook: it is a self-description written for machines that reads well for people too. New visitors, distribution channels beyond search engines, and your own content audits can all use it as a guide page — which also shapes how it should be written, as Section 4 covers.

3. File Format

The specification keeps the structure tight: a few parts in a fixed order, only one of which is required.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# Site Name

> A one-line summary of what this site is

Additional prose covering positioning, language, scope, update cadence

## Section Heading

- [Link title](https://example.com/page): what this link covers

## Optional

- [Secondary link](https://example.com/other): the model may or may not read this

What each part does, and its constraints:

A few hard requirements:

• Place it at the site root or at any subpath — /llms.txt or /docs/llms.txt. A file covers only the URLs under its own path, and when several apply, the client uses the most specific one
• It must be Markdown, served as plain text with a text/plain or text/markdown Content-Type
• A list item without a link is not a link — the spec calls for standard Markdown hyperlink syntax, not bare URLs
• Every part except the H1 is optional, but an llms.txt with only an H1 is close to worthless

3.1 Markdown Versions: the .md Suffix

Beyond the index file, the specification raises one more thing: give each page a clean Markdown source of its own, so the model can skip a round of HTML body extraction.

The examples below use this blog, but the site has not implemented this capability yet — they show what it would look like once it does.

The proposal accepts two URL forms:

• Append: page.html → page.html.md
• Replace the extension: page.html → page.md

A path with no filename uses index.html.md or index.md. The proposal itself uses the append form — by_example.html alongside by_example.html.md — and it is the better choice in practice: on static hosting, page.md can collide with a real route, while an appended suffix cannot.

1
2

curl -s https://www.chenshaowen.com/en/blog/rdma-ops.html      # default: HTML
curl -s https://www.chenshaowen.com/en/blog/rdma-ops.html.md   # append .md: Markdown

The latter returns the post’s source, with no template noise:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

---
title: RDMA Operations
tags:
  - RDMA
  - RoCE
  - InfiniBand
  - Network
  - Operations
  - High Performance Computing
enableToc: true
layout: post
url: blog/rdma-ops.html
updated: 2026-08-22 00:00:00
date: 2026-08-22 00:00:00
---

## 1. RoCE

### 1.1 Connection Requirements

RDMA requires the same network class end to end — the same /24, for instance.
...

Compare the same content as HTML — the model has to work out which block is the body:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

<body>
  <nav class="navbar">...</nav>
  <aside class="sidebar">...</aside>
  <main>
    <article>
      <h2>1. RoCE</h2>
      <h3>1.1 Connection Requirements</h3>
      <p>RDMA requires the same network class end to end...</p>
    </article>
  </main>
  <div class="adsense">...</div>
  <script>
    ...
  </script>
</body>

3.2 How the Model Finds That Markdown

Once a second file exists, how does a model know? The specification’s answer is standard link relations, in either of two equivalent forms:

1
2

<link rel="alternate" type="text/markdown" href="/en/blog/rdma-ops.html.md">
<link rel="describedby" href="/llms.txt">

Or in an HTTP response header, to the same effect:

Link: </en/blog/rdma-ops.html.md>; rel="alternate"; type="text/markdown",
      </llms.txt>; rel="describedby"

rel="alternate" type="text/markdown" points at that page’s Markdown version, and rel="describedby" points at the llms.txt covering it. The specification prefers the header form: it works for non-HTML resources too, and it can be added once in the web server or CDN configuration without touching any page.

3.3 The Alternative: Content Negotiation

There is another approach that leaves the URL unchanged and distinguishes the format by Accept request header — content negotiation, in HTTP terms. The specification does not mention it; it is a common alternative in engineering practice.

1
2
3
4
5
6
7
8

# Default: HTML
curl -s https://www.chenshaowen.com/en/blog/rdma-ops.html | head -2
# <!DOCTYPE html>

# Declare a preference: same URL, Markdown
curl -s -H 'Accept: text/markdown' \
     https://www.chenshaowen.com/en/blog/rdma-ops.html | head -2
# ---

By HTTP semantics, Accept expresses a preference rather than a command, so the more common form carries a weight:

Accept: text/markdown, text/html;q=0.9

That reads “Markdown preferred, HTML acceptable.” When the server cannot satisfy the request it should fall back to HTML rather than returning 406 — otherwise ordinary browsers would fail to load the page too.

The server-side decision looks roughly like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

map $http_accept $prefer_markdown {
    default            0;
    "~*text/markdown"  1;
}

location /blog/ {
    if ($prefer_markdown) {
        rewrite ^(/blog/.+)\.html$ $1.md last;
    }
}

This configuration only does string matching and does not parse the q weights; a real implementation needs to be more careful.

3.4 A Community Convention: llms-full.txt

Practice has also popularized a companion file, /llms-full.txt, which inlines the full text of every page in the list. To be clear: it is not in the specification — it is a community convention, supported by individual documentation platforms and tools.

The division of labor is this:

• llms.txt is the index: small and sharp, for choosing pages
• llms-full.txt is the corpus: large and complete, the whole thing in one request

For documentation sites and API references, where the total volume is bounded, llms-full.txt is worth more — the model gets everything in one request instead of several rounds of fetching. For a blog that keeps growing, its size runs away, and the index model fits better.

3.5 What It Costs

Giving pages Markdown versions is more fundamental than llms.txt — llms.txt answers “which one to read,” this answers “how to read it cleanly.” But it costs more:

The easiest trap is caching in the content-negotiation approach: if responses sit behind a CDN, the first request caches the HTML, and later requests carrying Accept: text/markdown get that same HTML — something the caller usually never notices. The cache key has to include Accept to be safe.

For that reason, several CDN vendors have started offering the conversion as an edge capability, dropping the site’s cost from “change the server” to “flip a switch.” These capabilities are still evolving; check each vendor’s documentation for what is actually supported.

4. How to Write It

The format takes five minutes to learn; the content is the hard part. The most common mistake in writing llms.txt is turning it into a sitemap in Markdown.

4.1 Curate, Don’t Enumerate

If a site has 700 posts, listing them all is the same as listing none — the model still has to read 700 lines to decide, and cannot tell major from minor.

A good llms.txt is an act of editing: pick the 10 to 30 pages genuinely worth using as entry points, and group them by theme.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

## Storage

- [Ceph Architecture and Operations](https://www.chenshaowen.com/en/blog/container-deploy-ceph-architecture-operations-and-testing.html): cluster architecture, routine inspection, pool and OSD management, monitoring, and upgrades
- [MinIO Multi-Node, Multi-Disk Deployment and Operations](https://www.chenshaowen.com/en/blog/minio-multi-node-multi-disk-deployment-and-maintenance.html): deployment, bucket and user management, backup, plus node reboots, single-disk replacement, and node rebuilds
- [JuiceFS Performance Testing](https://www.chenshaowen.com/en/blog/performance-testing-and-comparison-of-juicefs-ce-ee-and-dragonfly.html): measured dd / benchmark / fio comparisons across local disks, the community edition, the enterprise edition, and Dragonfly integration

## AI Infrastructure

- [Common GPU Operations and Fault Handling](https://www.chenshaowen.com/en/blog/common-gpu-operation-and-fault-handling.html): notes on XID error codes, dropped cards, ECC, memory leaks, and other faults, updated continuously
- [RDMA Operations](https://www.chenshaowen.com/en/blog/rdma-ops.html): connection requirements, installation, and point-to-point testing for RoCE, Soft RoCE, and InfiniBand

4.2 Every Description Must Answer “What Do I Get From Reading It”

A description is not a restatement of the title — it is a filtering criterion. Compare:

4.3 The Summary Should Say What It Is, Not That It Is Good

The summary is the part most easily written as marketing copy. “Dedicated to sharing high-quality technical content” carries no information for a model.

A useful summary has three elements: domain, form, and scale or boundary.

1
2

> A personal engineering blog about AI, cloud native, and infrastructure,
> with 700+ hands-on notes, every post pairing reproducible commands with conclusions.

4.4 Optional Is Only for Secondary Content

## Optional is the one section in the specification with a conventional meaning, holding nice-to-have material — the about page, changelogs, blogrolls, legal notices. Earlier versions of the proposal gave it mechanical semantics (skip the section when context is tight); the current version has dropped that rule, but keeping secondary material separate is still a useful convention.

Conversely, putting core content in Optional tells the model “you can skip this.”

5. Where It Stands, and the Argument Against It

One thing has to be said plainly here: llms.txt has no force, and no major vendor has committed to supporting it.

A few observations:

• No vendor backing. OpenAI, Google, Anthropic, and others have all stopped short of promising they will read llms.txt. It is not the kind of widely honored convention robots.txt is — robots.txt has been in use since 1994, comes with compliance pressure and a defined crawler behavior spec, and llms.txt has none of that.
• Public crawler-log analyses generally show that requests for /llms.txt from AI crawlers are rare, with traffic still concentrated on pages and a handful of feeds.
• But it is not useless. The cost is one static file and half an hour of organizing, and the downside is close to zero. That is its most practical use — a guide page and content audit for people to read.

So the honest positioning is: a cheap bet, not an SEO silver bullet.

If your goal is simply “make my content easier for AI to use,” then ranked by return on effort, the things actually worth doing first are these:

In other words: llms.txt is the icing. If the body text itself cannot be crawled, no llms.txt will save you.

6. Summary

What llms.txt asks for is simple: put one Markdown file at the root of your site and state in plain language what the site is and which content is worth reading.

Concretely:

• Format: the H1 is the only required element, followed by a blockquote summary, body prose, and Markdown link lists grouped by theme, with ## Optional for secondary material. Besides the root, it can live at any subpath, covering only the pages beneath it.
• Companion: give pages a .md Markdown version and declare it with rel="alternate" in a Link response header — a step more fundamental than llms.txt itself.
• How to write it: curate rather than enumerate, 10 to 30 selected entry points, each description stating value rather than restating the title.
• How to ship it: on a static site, drop the file at the site root — no server changes needed. To cover more pages, generate a first draft with a build tool and trim it by hand.
• How to think about it: no vendor backing and low crawler volume, but the cost is minimal and the downside small. Make sure the body text is crawlable first, then add this file in passing.

Half an hour gets you a clear self-description of your site — which is worth something on its own.

7. References

• The llms.txt specification
• AnswerDotAI/llms-txt