What Is llms.txt?

What is an llms.txt file for?

An llms.txt file gives AI agents a short, curated guide to a website: the site's name, a summary and links to the pages that matter, ideally in Markdown. Agents read it on demand while answering a user, then follow the links they need instead of parsing full HTML pages.

An AI agent is a program built on a large language model that fetches web pages while carrying out a task, such as a coding assistant reading library documentation. HTML pages wrap their content in navigation, scripts and ads, and the proposal notes that converting them “back into clean text is difficult and imprecise”. Context windows, the amount of text a model reads at once, “are still too small for most websites in their entirety”.

llms.txt answers both problems with one small file. The proposal expects agents “to view or search llms.txt to find the information they need, then follow the relevant links”, which point to clean Markdown versions of pages. The file stays small; the detail lives behind the links. The proposal states that the file was expected to be “mainly useful for inference rather than training, and that is how it has been used”.

Who proposed llms.txt, and when?

Jeremy Howard proposed llms.txt on September 3, 2024, and published version 2 of the proposal in August 2026. The specification is an informal community proposal hosted at llmstxt.org and on GitHub under AnswerDotAI, not a standard from a standards body.

Two dates mark its history. On 3 September 2024, Jeremy Howard of Answer.AI published “The /llms.txt file” as a proposal for sites to offer LLM-friendly content. In August 2026, version 2 revised the text after, in the author’s words, “two years of adoption”. The GitHub repository AnswerDotAI/llms-txt hosts the text for version control and public discussion, and the proposal describes itself as “open for community input”.

What does an llms.txt file look like?

An llms.txt file is plain Markdown that opens with an H1 name, the only required element, followed by a blockquote summary, optional notes without headings, and H2 sections that list links as [name](url): notes. An H2 named "Optional" holds links an agent can skip.

The proposal’s own example is a cut-down version of the file for FastHTML, a Python web library. Shortened further here, from llmstxt.org:

# FastHTML

> FastHTML is a python library which brings together Starlette, Uvicorn, HTMX, and fastcore's `FT` "FastTags" into a library for creating server-rendered hypermedia applications.

Important notes:

- Although parts of its API are inspired by FastAPI, it is *not* compatible with FastAPI syntax and is not targeted at creating API services

## Docs

- [FastHTML quick start](https://fastht.ml/docs/tutorials/quickstart_for_web_devs.html.md): A brief overview of many FastHTML features

## Optional

- [Starlette full documentation](https://gist.githubusercontent.com/jph00/809e4a4808d4510be0e3dc9565e9cbd3/raw/9b717589ca44cedc8aaf00b2b8cacef922964c0f/starlette-sml.md): A subset of the Starlette documentation useful for FastHTML development.

The 5 parts, in the order the proposal fixes:

1. Byte order mark: optional, invisible, at the very start.
2. H1 name: # FastHTML, the only required part.
3. Blockquote summary: the > line with the key facts needed to read the rest.
4. Details: any Markdown except headings, here the “Important notes” list.
5. File lists: H2 sections such as ## Docs, each a list of [name](url) links with optional notes after a colon.

The format is Markdown so that language models read it easily, and precise enough that the proposal expects “parsers and regex” to process it.

Where does the file live, and what does it cover?

The file sits at /llms.txt for a whole site or at any subpath, such as /docs/llms.txt. The proposal states that “a file covers the URLs under its path, and where more than one file applies, agents should use the most specific one.” A site with both files serves documentation readers /docs/llms.txt and everyone else /llms.txt. The FastHTML project places its file at /docs/ to cover only its documentation pages.

What changed in version 2 of the proposal?

Version 2, published in August 2026, answers how agents find the file and defines files in subfolders. Pages point to their llms.txt with rel="describedby", both page.html.md and page.md count as markdown versions, the most specific file applies, and context-expansion tooling is dropped.

The changes page on llmstxt.org gives a reason for each revision:

The changelog ties each change to adoption: user requests, divergent practice or behaviour v1 left undefined. The discovery links can be sent as HTML <link> elements or as an HTTP Link: header, which also works for the Markdown files themselves.

How is llms.txt different from robots.txt and sitemap.xml?

robots.txt tells crawlers which paths they are allowed to fetch, sitemap.xml lists every indexable page, and llms.txt offers a short curated overview for agents. llms.txt grants and blocks nothing; a crawler refused by robots.txt stays refused whatever llms.txt says.

Only robots.txt controls access. The proposal lists 3 reasons a sitemap does not replace llms.txt: it often lacks LLM-readable versions of pages, it excludes external URLs that help explain the site, and it usually covers far more text than fits in a context window. The proposal also explains why llms.txt is not a well-known URI under RFC 8615: well-known URIs exist only at the origin root, while many authors control only a path, such as a GitHub Pages project site. Crawler permissions remain a robots.txt matter; run a robots.txt check for AI bots to see which crawlers a file lets in.

Who publishes llms.txt files today?

The proposal names OpenAI, Anthropic and Google's Gemini team as publishers of llms.txt files for their developer documentation. Mintlify, GitBook, Yoast SEO, AIOSEO and Wix generate the file automatically, and the proposal states that thousands of sites publish one.

Publishers named on llmstxt.org:

• OpenAI at developers.openai.com/llms.txt
• Anthropic at docs.anthropic.com/llms.txt
• Google’s Gemini API docs at ai.google.dev/gemini-api/docs/llms.txt

Platforms that generate the file automatically, per the proposal’s integrations list:

• Mintlify, a documentation platform, for every site it hosts
• GitBook for published documentation sites
• Yoast SEO, a WordPress plugin
• AIOSEO, a WordPress plugin
• Wix for every Wix site

Ahrefs reported figures from its own study in an article updated 15 June 2026: 28% of 137,000 domains in Ahrefs Web Analytics publish an llms.txt file, and 97% of the roughly 38,000 domains with a valid file received zero requests for it in May 2026. Ahrefs notes that its sample leans toward technically engaged site owners.

Do Google and the AI crawlers read llms.txt?

Google states that no AI text files are needed to appear in AI Overviews or AI Mode, and OpenAI, Anthropic and Perplexity do not document any crawler that reads llms.txt. Chrome's Lighthouse checks the file but marks a missing one "not applicable".

Google’s guide to generative AI features in Search, updated 10 July 2026, is direct: “You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities).” It adds that publishing llms.txt for other services “will neither harm nor help your site’s visibility or rankings in Google Search”.

Chrome’s Lighthouse includes an llms.txt audit in its agentic-browsing checks. It flags a server error when fetching the file, and a 404 is marked “Not Applicable (N/A), as providing the file is optional at the moment”, per the audit page updated 5 May 2026.

The crawler documentation of OpenAI, Anthropic and Perplexity, checked on 25 September 2026, contains no statement that any of their crawlers requests llms.txt. OpenAI’s documentation site does publish its own llms.txt index for readers and agents. Bikash has a public experiment in the plan, “Does llms.txt change how AI crawlers behave?”: server logs for 30 days before and 30 days after publishing a file, split by user agent. Its status is planned, with no results yet; it is listed on the experiments page. Which bots exist, and what each one fetches, is set out in the directory of AI crawler user agents.

Evidence is thin. What does publishing one cost?

Is an llms.txt file worth publishing?

An llms.txt file is worth publishing for sites whose users rely on coding or research agents, such as software documentation, and it costs one small text file. It is not a lever for Google AI Overviews citations, according to Google's own guidance.

The proposal states that llms.txt files “are used most heavily for software documentation, where coding agents follow them to find API references and tutorials”, and names other uses: a business outlining its structure and policies, a personal site answering questions about a CV, a school sharing course information. The cost is a text file of a few kilobytes, updated when key pages change. Build one with the free llms.txt generator, which follows the v2 format and validates each line.