# llms.txt — history research, source material Research pass for the learning-register article (**exhibit thirteen**, strata2signal research hub). All facts below were gathered by live WebSearch/WebFetch on **2026-08-13**. Nothing here is written from model memory. Where a claim could not be verified against a primary source it is marked **UNVERIFIED** and left un-smoothed. > *Header note, corrected 2026-08-16 after publication:* this line previously read **exhibit ten**, > the number this research carried during the design era, before the article was scheduled. It ships > in exhibit **thirteen**'s kit; exhibit ten is *How we work*. Only the header number changed — > no source, date, quote or verification status below was touched. Companion file: `spec-snapshot.md` — pinned capture of llmstxt.org as of 2026-08-13. **Reading conventions** - Every bullet carries `[source: URL, fetched 2026-08-13, "quote"]`. - Quotes are ≤25 words and are as returned by the fetch. Fetches are HTML→markdown conversions by a summarising model, so **treat quotes as near-verbatim, not court-record verbatim** — re-fetch and confirm any sentence before publishing it inside quotation marks. - Numbered sources are listed at the end. **Standing caveat on the secondary literature.** llms.txt is a heavily SEO-farmed topic. A large fraction of search results are low-quality content-marketing pages that confidently assert unsourced specifics, including at least one fabricated standards event (see §5.2). Prefer §1 and §3 primary sources. Anything sourced only to an SEO blog is flagged as such. --- ## 1. Origin — who, when, where - llms.txt was proposed by **Jeremy Howard**, co-founder of Answer.AI and fast.ai, and published **3 September 2024**, simultaneously on the Answer.AI blog and at llmstxt.org. [source: https://www.answer.ai/posts/2024-09-03-llmstxt.html, fetched 2026-08-13, "Author: Jeremy Howard / Published: September 3, 2024"] — [S1] - The framing sentence of the original post is about a shift in *who* websites serve: [source: https://www.answer.ai/posts/2024-09-03-llmstxt.html, fetched 2026-08-13, "Today websites are not just used to provide information to people, but they are also used to provide information to large language models."] — [S1] - The original post argues LLM-directed content is a distinct craft: [source: https://www.answer.ai/posts/2024-09-03-llmstxt.html, fetched 2026-08-13, "Providing information for language models is a little different to providing information for humans, although there is plenty of overlap."] — [S1] - The core proposal sentence, v1: [source: https://www.answer.ai/posts/2024-09-03-llmstxt.html, fetched 2026-08-13, "We propose that those interested in providing LLM-friendly content add a `/llms.txt` file to their site."] — [S1] - The companion proposal — markdown twins of pages — v1 form, `.md` **appended**: [source: https://www.answer.ai/posts/2024-09-03-llmstxt.html, fetched 2026-08-13, "provide a clean markdown version of those pages at the same URL as the original page, but with `.md` appended."] — [S1] - The stated problem the proposal solves is **ambiguity of context construction**, and the argument that the site author is the right party to resolve it: [source: https://www.answer.ai/posts/2024-09-03-llmstxt.html, fetched 2026-08-13, "constructing the right context for LLMs based on a website is ambiguous"] — [S1] - **Motivating anecdote (useful colour, but secondary-sourced).** Howard had just released FastHTML, and the recurring complaint was that AI coding assistants could not help developers use it. [source: search summary aggregating llms-txt.io / codersera coverage, fetched 2026-08-13, "the most common complaint from developers was that AI coding assistants couldn't help them use it"] — **PARTIALLY VERIFIED**: the FastHTML linkage is unambiguous in the primary sources (FastHTML is the worked example in both v1 and v2), but the "most common complaint" causal story appears only in secondary write-ups. Do not quote it as Howard's own words without finding a first-person statement. — [S15] ### 1.1 What the spec actually specifies Full text in `spec-snapshot.md`. Structural summary: - Required and optional sections, in fixed order: optional BOM; **an H1 with the project or site name — the only required section**; a blockquote summary; zero or more non-heading markdown sections; zero or more H2-delimited "file lists" of markdown hyperlinks with optional `: notes`. [source: https://llmstxt.org/, fetched 2026-08-13, "An H1 with the name of the project or site. This is the only required section"] — [S2] - The deliberate choice of markdown over XML, and the dual-audience justification: [source: https://llmstxt.org/, fetched 2026-08-13, "it uses Markdown to structure the information rather than a classic structured format such as XML... we expect many of these files to be read by language models and agents"] — [S2] - Machine-parseable despite being markdown — the point the author is careful to make twice: [source: https://llmstxt.org/, fetched 2026-08-13, "human and LLM readable, but is also in a precise format allowing fixed processing methods (i.e. classical programming techniques such as parsers and regex)"] — [S2] - **The context-window reasoning (v2 wording).** This is the passage the article most needs: [source: https://llmstxt.org/, fetched 2026-08-13, "Context windows, while larger than they were, are still too small for most websites in their entirety, and every wasted token costs time and money."] — [S2] - **The crawling-vs-curation reasoning.** Why HTML is the wrong input: [source: https://llmstxt.org/, fetched 2026-08-13, "An HTML page wraps its information in navigation, ads, and JavaScript, and converting it back into clean text is difficult and imprecise."] — [S2] - **The index-not-payload design.** The file stays small; detail lives behind links: [source: https://llmstxt.org/, fetched 2026-08-13, "The file itself stays small enough to fit in context. The detail lives behind the links, and is fetched only when needed."] — [S2] - **Inference, not training** — the intended use, and the author's claim it played out: [source: https://llmstxt.org/, fetched 2026-08-13, "Our expectation was that llms.txt would mainly be useful for _inference_ rather than _training_, and that is how it has been used"] — [S2] ### 1.2 llms.txt vs llms-full.txt — an important correction - **`llms-full.txt` is NOT in the spec.** The v2 text at llmstxt.org contains no mention of `llms-full.txt`, `llms-ctx.txt`, `llms-ctx-full.txt`, or the `llms_txt2ctx` tool. Verified by full-text fetch of the spec and of the canonical GitHub repo. [source: https://github.com/AnswerDotAI/llms-txt, fetched 2026-08-13, "The provided content contains no references to llms-ctx.txt, llms-ctx-full.txt, llms-full.txt, or an llms_txt2ctx tool."] — [S2][S3] - **v1 did have a two-file derivative scheme, but under different names.** The original September 2024 post describes generated context files: [source: https://www.answer.ai/posts/2024-09-03-llmstxt.html, fetched 2026-08-13, "The two files are: llms-ctx.txt, which does not include the optional URLs, and llms-ctx-full.txt, which does include them. They are created using the `llms_txt2ctx` command line application."] — [S1] Note the naming: **`llms-ctx-full.txt`, not `llms-full.txt`.** These are different things, and the v2 Changes page confirms the `llms_txt2ctx` tooling was dropped. — [S4] - **`llms-full.txt` is a platform convention, originating with Mintlify.** Mintlify's own docs describe generating both, and describe the distinction: [source: https://www.mintlify.com/docs/ai/llmstxt, fetched 2026-08-13, "Mintlify automatically hosts an `llms-full.txt` file at the root of your project"] — [S5] - The working distinction, as stated in Mintlify-adjacent material: llms.txt is a curated index of links; llms-full.txt is the entire documentation concatenated into one file for a single-fetch ingest. [source: search summary over mintlify.com material, fetched 2026-08-13, "llms.txt is a curated map, llms-full.txt is a bulk dump"] — [S6] - **UNVERIFIED:** A widely-repeated claim that "the llms-full.txt file structure was developed by Mintlify in collaboration with customer Anthropic, and subsequently included as part of the official llms.txt proposal." This surfaced in a search-result summary. **I could not confirm it.** Mintlify's own blog post [S7] and its own docs page [S5] do not state it; the Mintlify blog credits Howard for llms.txt and mentions Anthropic only as an example implementer ("If you're interested in future-proofing your docs like Anthropic"). The tail of the claim — "included as part of the official proposal" — is affirmatively **contradicted** by the spec text, which never mentions llms-full.txt. Do not repeat this claim. - Mintlify's llms.txt blog post is dated **20 November 2024**, i.e. ~11 weeks after the proposal — useful for dating the platform-adoption wave. [source: https://www.mintlify.com/blog/simplifying-docs-with-llms-txt, fetched 2026-08-13, "Publication Date: November 20, 2024"] — [S7] --- ## 2. Prior art the proposal consciously echoes The spec names its antecedents explicitly rather than by implication: [source: https://llmstxt.org/, fetched 2026-08-13, "The approach of using a standard filename follows `/robots.txt` and `/sitemap.xml` at the site root."] — [S2] - **robots.txt (1994).** Created by Martijn Koster in 1994; formalised as an IETF Standards Track RFC only in September 2022, after ~28 years as a de facto standard. [source: https://datatracker.ietf.org/doc/html/rfc9309, fetched 2026-08-13, "the 'Robots Exclusion Protocol' method originally defined by Martijn Koster in 1994"] — [S8] Supporting detail: Koster proposed it on the www-talk mailing list in **February 1994** while at Nexor, after crawlers overwhelmed his own site. [source: search summary over Wikipedia/greenhills.co.uk, fetched 2026-08-13, "Koster proposed the standard while working for Nexor in February 1994 on the www-talk mailing list"] — **PARTIALLY VERIFIED**: the year and authorship are primary-sourced via RFC 9309 [S8]; the February date and the www-talk detail are secondary. Koster's own retrospective page (greenhills.co.uk) returned HTTP 403 and could not be fetched. - The spec is careful that the echo is *formal, not functional* — it distinguishes purpose: [source: https://llmstxt.org/, fetched 2026-08-13, "robots.txt lets automated tools know what access to a site is considered acceptable... llms.txt information is instead used on demand"] — [S2] - **sitemap.xml (2005–2006).** Google launched the Sitemaps 0.84 protocol in 2005 and published it under a Creative Commons Attribution/Share-Alike licence to encourage adoption; Google, Yahoo! and Microsoft announced joint support in **November 2006**, jointly sponsoring sitemaps.org. [source: search summary over en.wikipedia.org/wiki/Sitemaps and lorelle.wordpress.com, fetched 2026-08-13, "Google, Yahoo! and Microsoft announced joint support for the Sitemaps protocol in November 2006"] — [S9] — **PARTIALLY VERIFIED**: widely and consistently attested, but I did not fetch a primary Google announcement. Low risk. - The spec's own argument for why sitemap.xml is not a substitute — three reasons, worth citing in full because it is the clearest statement of what llms.txt is *for*: [source: https://llmstxt.org/, fetched 2026-08-13, sitemap.xml "Often won't have the LLM-readable versions of pages listed... Doesn't include URLs to external sites... will be too large to fit in an LLM context window"] — [S2] - **humans.txt (2011).** A Spanish web-developer initiative to credit the people behind a site; a deliberate counterpart to robots.txt. Tagline and self-description from the initiative's own site: [source: https://humanstxt.org/, fetched 2026-08-13, "It's an initiative for knowing the people behind a website. It's a TXT file that contains information about the different people who have contributed"] and ["We Are People, Not Machines."] — [S10] **UNVERIFIED (date):** humanstxt.org itself does not state a launch date on the fetched page. Secondary sources consistently say **2011** and name Carlos Mañas, Maria Macias, Abel Cabans, Abel Sutilo and Juanjo Bernabeu. [source: search summary over creativebloq.com/humanstxt.org, fetched 2026-08-13, "The initiative launched in 2011"] — [S11]. Treat 2011 as well-attested-but-secondary; the site itself credits Abel Cabans with setting the format ("set the grounds in his own humans.txt file"). **Note:** the spec does *not* name humans.txt as an antecedent — that connection is the article's own reading, and should be presented as such, not attributed to Howard. - **The `/.well-known/` road not taken (RFC 8615).** New in v2, and the most substantive design-reasoning passage available. The spec rejects `/.well-known/` because it only exists at the origin root, whereas many authors control only a path: [source: https://llmstxt.org/, fetched 2026-08-13, "a GitHub Pages project site, for example, can publish files in its own directory but can never add one to the host's `/.well-known/`"] — [S2] This objection was raised in the W3C strategy issue two years earlier — see §5.1. --- ## 3. Adoption — verified live, 2026-08-13 ### 3.1 AI labs (spec claims these three; all three verified live) The spec asserts: [source: https://llmstxt.org/, fetched 2026-08-13, "The AI labs themselves publish llms.txt files for their own developer docs: OpenAI, Anthropic, and Gemini."] — [S2] I checked each independently: - **OpenAI — VERIFIED LIVE.** `https://developers.openai.com/api/llms.txt` returns a spec-conformant file: H1 `# OpenAI API`, blockquote, `## Documentation sets` and `## Optional` sections with markdown links. [source: https://developers.openai.com/api/llms.txt, fetched 2026-08-13, "Routing index for OpenAI API guides, endpoint reference, and common implementation tasks."] — [S12] Note: `https://platform.openai.com/llms.txt` returns **404** — the file is under `/docs/llms.txt` and on the developers subdomain, not at the platform root. Use the developers.openai.com URL in the article. OpenAI additionally publishes segmented files at `https://cdn.openai.com/API/docs/txt/` including `llms-full.txt`. - **Anthropic — VERIFIED LIVE.** `https://docs.anthropic.com/llms.txt` 301-redirects to `https://platform.claude.com/docs/llms.txt`, which serves a valid file beginning `# Anthropic Developer Documentation`. It is an interesting non-canonical variant: it uses H2 sections for `## Root URL` and `## Available Languages on Website` and enumerates per-language page counts (English 565 pages; nine other languages at 206 pages each) rather than being a pure link index. [source: https://platform.claude.com/docs/llms.txt, fetched 2026-08-13, "# Anthropic Developer Documentation"] — [S13] **Cite the redirect target, not docs.anthropic.com**, or the article's link will bounce. - **Google / Gemini — VERIFIED LIVE.** `https://ai.google.dev/gemini-api/docs/llms.txt` serves a valid file: H1 `# Gemini API`, description line, `## Docs` section with links to `.md.txt` page variants. [source: https://ai.google.dev/gemini-api/docs/llms.txt, fetched 2026-08-13, "# Gemini API / Gemini API Docs and API Reference / ## Docs"] — [S14] **This is a good irony hook for the article:** Google Search's official guidance says llms.txt is not used (§4.1), while Google's own Gemini developer docs publish one, and Chrome's Lighthouse audits for one (§3.3). - **Cloudflare — VERIFIED LIVE**, and architecturally interesting: the root file is an index *of other llms.txt files*, one per product, demonstrating the v2 subpath-scoping model in the wild. [source: https://developers.cloudflare.com/llms.txt, fetched 2026-08-13, "Each product below links to its own llms.txt, which contains a full index of that product's documentation pages"] — [S16] ### 3.2 Documentation platforms and CMSs that auto-generate Listed by the spec itself, each with a vendor doc page [S2]: **Mintlify** (generates llms.txt plus markdown page versions for every hosted site), **GitBook**, **Yoast SEO** (WordPress), **AIOSEO** (WordPress), **Wix** (every Wix site). - Mintlify additionally serves both files under `/.well-known/` for tool compatibility — i.e. platforms are hedging against the spec's own `/.well-known/` rejection. [source: https://www.mintlify.com/docs/ai/llmstxt, fetched 2026-08-13, "Both files are also hosted at `/.well-known/` locations for tool compatibility."] — [S5] - **Shopify mass-deployment, ~late April–May 2026.** The single largest adoption event, and the one that most distorts adoption statistics. Shopify auto-deployed llms.txt across its storefronts with no merchant opt-in or notification. [source: search summary over craftshift.com / nikhil.pro / weaverse.io, fetched 2026-08-13, "Shopify silently deployed an auto-generated llms.txt to all 7 million storefronts in late April and early May 2026"] — [S17] — **SECONDARY-SOURCED ONLY.** I did not find a primary Shopify changelog entry. The corroborating independent signal is Casey Burridge's HTTP Archive analysis, which finds Shopify at **78.1%** llms.txt adoption vs WordPress at 8.7% [S18] — a platform-default signature. Treat the "7 million" figure as unconfirmed; the *fact* of a platform default is well-corroborated. ### 3.3 Chrome Lighthouse — the audit - Lighthouse **13.3** added an "Agentic Browsing" category including an llms.txt audit. [source: https://www.searchenginejournal.com/googles-llms-txt-guidance-depends-on-which-product-you-ask/575431/, fetched 2026-08-13, "Lighthouse 13.3 now includes an llms.txt audit"] — [S19] - The audit's own documentation is careful and deflationary about status: [source: https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt, fetched 2026-08-13, "an emerging convention used to provide a machine-readable summary of a website's content, specifically designed for LLMs and AI agents"] — [S20] Page last-updated stamp: **2026-05-05 UTC**. - **Crucially, the audit does not penalise absence.** A 404 marks the audit Not Applicable: [source: https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt, fetched 2026-08-13, the file is "optional at the moment"; N/A if not found] — [S20] It fails only on a server error. Where the file exists, it checks for a missing H1, excessive brevity, and absence of links. The category is marked "under development" and produces a fractional pass ratio rather than a 0–100 score. - **Do not overstate this.** The spec says "Chrome's Lighthouse audits sites for one", which is true but reads as stronger endorsement than the audit's own N/A-on-absence behaviour supports. Worth noting the gap in the article. ### 3.4 Directories and registries - **directory.llmstxt.cloud** — created by @ifox and @joyceverheije; ~21 pages of listings. Notable entries include Anthropic, Perplexity, Cloudflare, Vercel, Coinbase, Hugging Face, Cursor, ElevenLabs, Zapier, Pinecone, Resend. No last-updated date shown. [source: https://directory.llmstxt.cloud/, fetched 2026-08-13, "websites and companies leading the adoption of the llms.txt standard"] — [S21] - **llmstxthub.com** — self-describes as the largest such directory; **2,650 websites** tracked at fetch time, across 15+ categories, with recent additions timestamped August 2026. Ships a Chrome extension, VS Code extension, MCP explorer, Raycast extension and CLI. [source: https://llmstxthub.com/, fetched 2026-08-13, "the largest directory for AI-ready documentation and tools implementing the proposed llms.txt standard"] — [S22] - **llmstxt.site** — tracks files with token-size measurements. Notable for showing how far the convention has drifted from "small enough to fit in context": entries range from 0 tokens to ~107.8M tokens. Typical range 500–50,000 tokens. [source: https://llmstxt.site/, fetched 2026-08-13, token sizes "0 tokens" to "~107.8M tokens"] — [S23] — **The 107.8M outlier is a strong article detail** (a file three orders of magnitude past any context window, defeating the spec's stated purpose), but verify it independently before publishing; it is a single directory's self-reported measurement. - All three directories are the ones the spec itself links [S2]. --- ## 4. The skeptical case — well-evidenced, and the article needs it ### 4.1 Google's position: repeatedly, publicly, no - **John Mueller, Bluesky, 17 June 2025** — the most-quoted line in the entire debate: [source: https://bsky.app/profile/johnmu.com/post/3lrshm4gggs2v via search summary and https://www.seroundtable.com/google-ai-llms-txt-39607.html, fetched 2026-08-13, "FWIW no AI system currently uses llms.txt."] — [S24][S25] **PARTIALLY VERIFIED:** the Bluesky permalink and the quote are consistently reported across multiple independent outlets and the post URL is public, but seroundtable.com returned **HTTP 403** to my fetch and I did not fetch bsky.app directly. Confirm the post before quoting, and cite the Bluesky permalink rather than an SEO blog. - Mueller's supporting empirical claim — the server-log argument: [source: search summary over seroundtable/searchenginejournal coverage, fetched 2026-08-13, "the consumer LLMs / chatbots... will fetch your pages - for training and grounding, but none of them fetch the llms.txt file"] — [S24] — **SECONDARY-SOURCED**, verify wording. - Mueller's keywords-meta-tag comparison, which is the sharpest form of the critique: [source: search summary over Reddit-quoting coverage, fetched 2026-08-13, "AFAIK none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it)"] and ["At that point, why not just check the site directly?"] — [S24] — **SECONDARY-SOURCED**, verify wording. - Mueller, later and more concessive — the pragmatic test: [source: https://www.searchenginejournal.com/google-says-llms-txt-is-purely-speculative-for-now/577576/, fetched 2026-08-13 (Roger Montti, 2 June 2026), "When an AI platform that brings you clients complains that it needs the file for your site, then I'd recommend taking the time to create one."] — [S26] - And on its epistemic status: [source: same, fetched 2026-08-13, "I don't think anyone knows – it's purely speculative for now (the file has existed for years, yet none of the AI systems use it — what does it mean?)."] — [S26] - **Google Search's official documentation (primary source, best citation available).** Last updated **2026-07-10 UTC**: [source: https://developers.google.com/search/docs/fundamentals/ai-optimization-guide, fetched 2026-08-13, "LLMS.txt files and other 'special' markup: You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them."] — [S27] And: ["Doing so will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them."] — [S27] This appears under a "Mythbusting generative AI search" heading. Google listed the change in its June 2026 documentation updates under "Clarifying guidance on llms.txt files." - **Google's internal contradiction is itself the story.** Search says ignore it; Chrome audits for it; Gemini's docs publish one. [source: https://www.searchenginejournal.com/googles-llms-txt-guidance-depends-on-which-product-you-ask/575431/, fetched 2026-08-13 (Matt G. Southern, 20 May 2026), "For Google Search, llms.txt isn't needed for AI Overviews, AI Mode, or other generative AI Search features. For browser-based agents, Lighthouse considers llms.txt optional."] — [S19] - Mueller's reconciliation of the two — a good, fair-minded quote: [source: https://searchengineland.com/google-llms-txt-chrome-lighthouse-478246, fetched 2026-08-13 (Danny Goodwin, 20 May 2026), "The short answer is that it's not done for search. There's more to websites than just SEO."] — [S28] And the memorable line: ["Making a markdown version of a shoe's specs is not going to get you more sales (competitors appreciate it tho)."] — [S28] - **Gary Illyes** reportedly confirmed at Search Central Live (July 2025) that Google does not support llms.txt and has no plans to. [source: search summary + Spriestersbach, fetched 2026-08-13, Google has "no plans to do so"] — [S29] — **UNVERIFIED**: conference remark, no primary transcript located. The Google documentation [S27] carries the same position with a citable primary source — **prefer [S27] and drop the Illyes attribution** unless a transcript is found. ### 4.2 The measurements — this is the strongest evidence, and it is damning - **Ahrefs, 15 June 2026** (Louise Linehan, with Xibeijia Guan). Methodology: 137,210 domains in Ahrefs Web Analytics with traffic in May 2026; each root checked for an HTTP-200 llms.txt validated as actual markdown (to exclude soft 404s); every request to `/llms.txt` classified by response code and user agent via Ahrefs Bot Analytics. [source: https://ahrefs.com/blog/llmstxt-study/, fetched 2026-08-13] — [S30] Headline figures, as stated: - "28% of domains publish llms.txt" - "97% of those files received zero traffic in May 2026" - "96% of the requests that did reach llms.txt files came from bots" - "19.5% of fetches came from named AI tools" - "77% of the bots reading llms.txt aren't from AI tools at all" - "Zero requests came from AI bots for llms.txt files that don't exist" **Two caveats the article should carry.** (a) The 28% publish rate is *not* a web-wide figure — the sample is domains using Ahrefs Web Analytics, i.e. SEO-instrumented sites, which massively over-represents adopters. Compare the 5.61%/8.7%/10.13% independent figures in §5.3. (b) The zero-requests-for-nonexistent-files finding is the analytically important one: it shows agents **do not probe** for llms.txt, so publishing one is only discoverable if something else points at it. - Ahrefs' interpretation of the mechanism — worth quoting, because it is more precise than "nobody uses it": [source: https://ahrefs.com/blog/llmstxt-study/, fetched 2026-08-13, "AI tools fetch llms.txt when a link, an index, or a user instruction tells them it exists"] — [S30] - **A detail with direct relevance to a Claude-Code-adjacent audience:** Ahrefs found Claude-Code outfetched every AI *retrieval* bot, despite retrieval bots being the presumed audience. [source: https://ahrefs.com/blog/llmstxt-study/, fetched 2026-08-13, "Claude-Code outfetched every AI retrieval bot"] — [S30] This supports the spec's own claim that the real constituency is **coding agents**, not search. - **Otterly.ai** — independent 90-day server-log experiment: of 62,100+ AI-bot visits, **84** targeted `/llms.txt` — **0.1%**. [source: https://otterly.ai/blog/the-llms-txt-experiment/ via search summary, fetched 2026-08-13, "just 84 requests targeted the /llms.txt file... just 0.1% of AI visits"] — [S31] Conclusion: "llms.txt is not a significant driver of AI crawl behavior." — **PARTIALLY VERIFIED**: figures consistent across two independent summaries (including Spriestersbach [S29]); I did not fetch the Otterly page directly. Do so before publishing the 84/62,100 figure. - **The self-referential finding** — the industry studying llms.txt generates more traffic to llms.txt than AI systems do. SEO audit tools were the largest single source of fetches (21.7%); training crawlers such as GPTBot were 4.51%. [source: https://ppc.land/llms-txt-adoption-rises-8-8x-but-97-of-files-get-zero-ai-requests/, fetched 2026-08-13 (Luis Rijo, 2 July 2026), requests "came primarily from SEO audit tools (21.7%), not from AI assistants for which llms.txt was designed"] — [S32] Related and vivid: Slackbot's link-preview crawler fetched llms.txt more often than PerplexityBot did. [source: search summary over ahrefs/otterly coverage, fetched 2026-08-13] — **UNVERIFIED**, attribute to a specific study before using. - **SE Ranking, 7 November 2025** — ~300,000 domains, testing whether llms.txt predicts AI citation, using Spearman correlation, XGBoost regression and SHAP: [source: https://seranking.com/blog/llms-txt/, fetched 2026-08-13, "having an LLMs.txt file didn't make a domain more likely to be cited by AI models"] — [S33] Predictive accuracy **improved** when the variable was removed. Adoption 10.13%, and strikingly flat across traffic tiers. Their bottom line: minimal technical risk, no measurable benefit today, possible future-proofing value. ### 4.3 The argument-from-principle critique - **Kai Spriestersbach, 23 February 2026** — the sharpest published statement of the case. [source: https://medium.com/@kaispriestersbach/the-llms-txt-is-dead-more-precisely-a-dud-ab7bee4f469c, fetched 2026-08-13] — [S29] Four arguments: (1) the Otterly 0.1% adoption data; (2) Google's stated position; (3) manipulation vulnerability — llms.txt is "a signal controlled by the site operator", directly analogous to the deprecated keywords meta tag; (4) purpose misalignment — Howard designed it for *developer documentation*, not general AI-search visibility. The load-bearing line: ["Search engines evaluate. They don't let themselves be evaluated."] - **Point (4) is the fairest and most important framing for the article.** Much of the "llms.txt is dead" literature measures llms.txt against a goal it never claimed — SEO/AI-search visibility — and the spec's own text supports the narrower claim: [source: https://llmstxt.org/, fetched 2026-08-13, "llms.txt files are used most heavily for software documentation, where coding agents follow them to find API references and tutorials."] — [S2] A learning-register article should separate "does it work for AI SEO?" (evidence: clearly no) from "does it work for coding agents fetching docs?" (evidence: partial — Ahrefs' Claude-Code finding [S30] is the only direct measurement I located, and it is a relative, not absolute, result). - Early reception was already skeptical in September 2024 — Hacker News commenters questioned whether LLMs needed the help and whether platforms would ever use it, with some concerned it eased unlicensed scraping. [source: search summary, fetched 2026-08-13] — **UNVERIFIED**: I could not locate the specific HN thread. Either find and cite the thread directly or drop this; do not characterise a discussion I have not read. - A contemporaneous *optimistic* take, for balance — Search Engine Land, 28 March 2025 (Rob Garner): [source: https://searchengineland.com/llms-txt-proposed-standard-453676, fetched 2026-08-13, "It definitely remains to be seen. llms.txt has a long road ahead, but I wouldn't bet against it."] — [S34] Same article, on status at the time: ["Are all the LLMs using it? It is definitely not even near critical mass, but some have reported an interest."] It also records dissent from Brett Tabke, who argued existing standards already sufficed. --- ## 5. Current state as of August 2026 ### 5.1 The spec itself evolved — v2, 10 August 2026 **This is the freshest and most under-reported fact in the research, six days before the article's date.** The spec was revised into a v2 and llmstxt.org now carries the title "The /llms.txt file, v2", modified 2026-08-10, still bylined Jeremy Howard, still showing the original 2024-09-03 publication date. [source: https://llmstxt.org/, fetched 2026-08-13] — [S2] Changes v1 → v2, from the spec's own Changes page [source: https://llmstxt.org/changes.html, fetched 2026-08-13] — [S4]: - **Discoverability via link relations** — the most-requested addition. `rel="alternate" type="text/markdown"` for a page's markdown twin; `rel="describedby"` for the llms.txt covering it; deliverable as HTML `` elements or an HTTP `Link:` header. The header form works for non-HTML resources and can be set at the CDN without touching pages. This is a **direct response to the §4.2 finding that agents never probe for the file** — v2 makes it announce itself. - **Both markdown URL forms permitted.** v1 mandated appending `.md`; real tooling replaced the extension instead; v2 allows both. A spec conceding to implementations. - **Subpath scoping defined.** A file covers pages under its path; most specific wins. This is what lets a GitHub Pages project participate, and it is the same argument the spec uses against `/.well-known/` (§2). - **Tooling dropped.** `llms_txt2ctx` and the ctx-file scheme are gone; "Optional" loses any mechanical meaning and survives as convention only. The v2 background paragraph is also a notable rhetorical shift — from prediction to claimed fact: [source: https://llmstxt.org/, fetched 2026-08-13, "When this proposal was first written in 2024, this was largely a prediction. Today it is routine."] — [S2] **UNVERIFIED / GAP:** I could not retrieve the **original v1 text of llmstxt.org** for a side-by-side comparison. `web.archive.org` is blocked by this environment's fetch tooling ("Claude Code is unable to fetch from web.archive.org"). The Answer.AI post [S1] is the surviving primary artefact of the v1 era and is the correct substitute, but note that it is the *blog post*, which differed in wording from the v1 spec site. If the article wants a genuine v1-vs-v2 diff, either retrieve the archive snapshot by another route or diff the git history of `github.com/AnswerDotAI/llms-txt` — the repo hosts the document under version control [S3] and is the cleanest available path. ### 5.2 Standardisation — no formal standard exists - **W3C: an open investigation, not a standard.** W3C strategy issue #506 proposes standardising llms.txt. Opened by **xfq on 27 April 2025**; still **open**, in the Investigation phase of the W3C Technical Strategy Pipeline, with no formal decision recorded. [source: https://github.com/w3c/strategy/issues/506, fetched 2026-08-13, status "Open (Investigation phase)"] — [S35] Concerns logged in the issue: the `/llms.txt` path may conflict with `/.well-known/`; large documentation sets can exceed context windows; adoption at the time was limited beyond the AnswerDotAI/fast.ai ecosystem. A May 2025 update noted more tools adopting it. Note the `/.well-known/` objection here predates and is answered by the v2 spec text (§2) — a clean narrative thread. - ⚠️ **UNVERIFIED AND PROBABLY FALSE — DO NOT REPEAT.** Multiple SEO-marketing pages claim the W3C published a working draft titled **"AI Crawler Guidance Standardization"** on **16 June 2026**, proposing to formalise llms.txt, with four specific changes (root-only location, a versioning header line, a strict markdown subset, and a normative llms.txt-overrides-robots.txt precedence rule for AI crawlers) and a 90-day comment period closing 14 September. [source: https://www.pravinkumar.co/blog/w3c-llms-txt-standard-proposal-june-2026-webflow-2026, fetched 2026-08-13 (Pravin Kumar, 23 June 2026)] — [S36] **Grounds for rejection:** (a) the article cites **no clickable link** to any W3C document, repository, or working group; (b) a domain-restricted search of w3.org and github.com returns **no document of that title**; (c) it is contradicted by the live state of W3C issue #506, which remains in Investigation with no draft [S35]; (d) the claimed "llms.txt takes precedence over robots.txt" rule is contrary to the actual spec's framing of the two as different in kind [S2]. **Flag this in the article if it discusses standardisation** — a fabricated standards event circulating in the SEO literature is itself a good exhibit for a learning-register piece about how conventions get mythologised. - **The real adjacent standards work is at IETF, and it is not llms.txt.** The IETF **AI Preferences (aipref)** working group is developing a vocabulary for expressing AI usage preferences (`draft-ietf-aipref-vocab`, revision 06, Proposed Standard track) defining categories such as `train-ai` and `search`, plus an attachment mechanism (`draft-ietf-aipref-attach`) defining a `Content-Usage` header and a robots.txt `Content-Usage` rule. [source: https://datatracker.ietf.org/doc/draft-ietf-aipref-vocab/ and https://datatracker.ietf.org/doc/draft-ietf-aipref-attach/ via search, fetched 2026-08-13, the vocab draft "does not yet have consensus"] — [S37] This is a **permissions** effort — the robots.txt lineage — not a **content-curation** effort like llms.txt. The two are complementary, not competing, and conflating them is a common error in the secondary literature. — **PARTIALLY VERIFIED**: draft names, numbers and status come from a search over datatracker; fetch the datatracker pages directly before citing revision numbers or milestone dates. - Bottom line, sourced: llms.txt remains **a community proposal maintained by its author in a GitHub repo** — no RFC, no W3C Recommendation, no vendor consortium. The only quasi-official institutional recognition is Chrome's Lighthouse audit (§3.3), which explicitly declines to penalise absence. ### 5.3 Adoption breadth — four measurements, four different numbers Report these as a **range with methodologies attached**, never as a single figure. The spread is itself the finding. | Measurement | Figure | Sample / method | Date | |---|---|---|---| | Casey Burridge / HTTP Archive [S18] | **5.61%** of top 10,000 (421 of 7,504 crawled) | HTTP Archive BigQuery, CrUX-origin crawl, `llms_txt_validation` field | June 2026 | | Rankability tracker [S38] | **8.7%** of top 1,000 (87 of 1,000) | Tranco top 1,000, monthly, HTTPS request for `/llms.txt` and `/llms-full.txt`, HTTP 200 + plain text required | June 2026 (page updated 2026-08-06) | | SE Ranking [S33] | **10.13%** of ~300,000 domains | ~300k indexed domains | Nov 2025 | | Ahrefs [S30] | **28%** of 137,210 domains | Domains using Ahrefs Web Analytics — **heavily SEO-skewed sample** | May 2026 | - **Growth is real and large regardless of level.** Burridge: 1.04% (July 2025) → 5.61% (June 2026), ~5.4x in twelve months. [source: https://caseyrb.com/blog/state-of-llms-txt-adoption/, fetched 2026-08-13 (Casey Burridge, 20 June 2026, co-written with an AI), "adoption increased from 1.04% in July 2025 to 5.61% by June 2026"] — [S18] Originality.ai, via PPC Land: 4,088 instances (June 2025) → 36,120 (May 2026), **8.8x** across 3M+ websites. [source: https://ppc.land/llms-txt-adoption-rises-8-8x-but-97-of-files-get-zero-ai-requests/, fetched 2026-08-13, "from 4,088 instances in June 2025 to 36,120 by May 2026—an 8.8x increase"] — [S32] - **Much of the growth is platform defaults, not decisions.** Burridge's platform split — Shopify **78.1%**, WordPress **8.7%** — separates automatic rollout from human adoption, and is the single most useful number in the whole adoption literature. [source: https://caseyrb.com/blog/state-of-llms-txt-adoption/, fetched 2026-08-13, "Shopify dominance at 78.1% due to automatic rollout, while WordPress adoption sits at 8.7%, reflecting genuine human adoption decisions"] — [S18] - **Adoption is flat across site tiers**, which argues against prestige-driven adoption: Burridge finds only 1.2 percentage points separating the top 1,000 from the top 1,000,000 [S18]; SE Ranking finds ~10% across low, mid and high traffic tiers [S33]. SE Ranking reportedly found **0%** among the top 1,000 by traffic, which conflicts with Rankability's 8.7% for the Tranco top 1,000 — different rankings and different dates, but note the conflict rather than picking a side. - Rankability's honest denominator note is worth borrowing methodologically: among the 549 *reachable* sites in its top-1,000 sample the rate is **15.8%**, because 451 were unreachable infrastructure/backend domains kept deliberately in the denominator. [source: https://www.rankability.com/data/llms-txt-adoption/, fetched 2026-08-13, "A site is counted as an adopter only when it returns a real text file (HTTP 200 with plain-text content)."] — [S38] ### 5.4 A successor convention is competing for the same slot - **agents.md.** Emerged late 2025, oriented to agent capabilities and contracts rather than content indexing. Shopify made `/agents.md` the canonical AI-discovery file for all storefronts around **20 May 2026**, with `/llms.txt` and `/llms-full.txt` reduced to mirroring/redirecting to it. [source: search summary over weaverse.io / craftshift.com / llms-txt.io, fetched 2026-08-13, "/agents.md is now the canonical AI-discovery surface on every Shopify store, with /llms.txt and /llms-full.txt defaulting to mirroring it"] — [S17][S39] — **SECONDARY-SOURCED ONLY**, no primary Shopify changelog located. Verify before publishing. A claimed 60,000+ open-source repositories use agents.md as of mid-2026 — **UNVERIFIED**. - **Note the reversal, if it holds:** the largest single source of llms.txt files on the web demoted llms.txt to a pointer within weeks of deploying it. That is a significant qualifier on every adoption statistic dated after May 2026, and on the spec's own v2 claim that "thousands of sites publish an llms.txt file." --- ## 6. Gaps and unverified items — consolidated Flagged so the author does not smooth over them: 1. **Original v1 llmstxt.org text — NOT RETRIEVED.** web.archive.org is blocked in this environment. Suggested route: git history of `github.com/AnswerDotAI/llms-txt` [S3]. 2. **llms-full.txt "Mintlify + Anthropic collaboration" — UNVERIFIED**, and its "included in the official proposal" tail is contradicted by the spec. Do not repeat. (§1.2) 3. **"June 2026 W3C working draft" — UNVERIFIED, probably fabricated.** No primary source exists; contradicted by live W3C issue #506. (§5.2) 4. **Gary Illyes' Search Central Live remark — UNVERIFIED** (no transcript). Use Google's documentation [S27] instead. (§4.1) 5. **Mueller's Bluesky post — PARTIALLY VERIFIED.** Quote is consistent across outlets; the bsky permalink was not fetched directly and seroundtable returned 403. (§4.1) 6. **Otterly.ai 84/62,100 figures — PARTIALLY VERIFIED** via two independent summaries; page not fetched directly. (§4.2) 7. **Shopify mass-rollout and the agents.md switch — SECONDARY-SOURCED ONLY.** No primary Shopify changelog found. The 7-million-storefront figure is unconfirmed; the platform-default *pattern* is corroborated by HTTP Archive data [S18]. (§3.2, §5.4) 8. **September 2024 Hacker News reception — UNVERIFIED.** Thread not located; either find it or cut the claim. (§4.3) 9. **humans.txt 2011 launch date — PARTIALLY VERIFIED.** humanstxt.org does not state it; secondary sources are consistent. (§2) 10. **robots.txt February 1994 / www-talk detail — PARTIALLY VERIFIED.** Year and authorship are primary via RFC 9309; the month and mailing-list detail are secondary. Koster's own retrospective page returned HTTP 403. (§2) 11. **llmstxt.site 107.8M-token outlier — single-source**, verify before using. (§3.4) 12. **Slackbot > PerplexityBot claim — UNVERIFIED**; attribute to a specific study or cut. (§4.2) 13. **IETF aipref draft revisions/milestones — PARTIALLY VERIFIED** via search over datatracker; fetch the datatracker pages directly before citing revision numbers. (§5.2) 14. **Ahrefs "28% publish" figure is sample-biased** and must not be presented as a web-wide adoption rate. (§4.2, §5.3) --- ## SOURCES All fetched 2026-08-13 unless noted otherwise. **Primary — spec and origin** 1. [S1] *"/llms.txt — a proposal to provide information to help LLMs use websites"*, Jeremy Howard, Answer.AI — published 2024-09-03. https://www.answer.ai/posts/2024-09-03-llmstxt.html 2. [S2] *"The /llms.txt file, v2"*, llmstxt.org — published 2024-09-03, modified 2026-08-10. https://llmstxt.org/ 3. [S3] AnswerDotAI/llms-txt GitHub repository (canonical spec repo; 2.6k stars, 159 forks). https://github.com/AnswerDotAI/llms-txt 4. [S4] *"Changes"* (v1→v2 rationale), llmstxt.org — August 2026. https://llmstxt.org/changes.html **Primary — platforms and adopters** 5. [S5] *"llms.txt"*, Mintlify documentation. https://www.mintlify.com/docs/ai/llmstxt 6. [S6] Mintlify blog / llms.txt vs llms-full.txt distinction (via search aggregation). https://www.mintlify.com/blog/what-is-llms-txt 7. [S7] *"Simplifying docs for AI with /llms.txt"*, Mintlify — 2024-11-20. https://www.mintlify.com/blog/simplifying-docs-with-llms-txt 8. [S12] OpenAI API llms.txt (live, verified). https://developers.openai.com/api/llms.txt 9. [S13] Anthropic developer docs llms.txt (live, verified; docs.anthropic.com 301s here). https://platform.claude.com/docs/llms.txt 10. [S14] Gemini API llms.txt (live, verified). https://ai.google.dev/gemini-api/docs/llms.txt 11. [S16] Cloudflare developer docs llms.txt (live, verified). https://developers.cloudflare.com/llms.txt **Primary — prior art** 12. [S8] RFC 9309, *Robots Exclusion Protocol* — IETF Standards Track, September 2022. https://datatracker.ietf.org/doc/html/rfc9309 13. [S9] *Sitemaps* — Wikipedia / contemporaneous 2006 coverage (secondary; see §2). https://en.wikipedia.org/wiki/Sitemaps 14. [S10] humanstxt.org — the initiative's own site. https://humanstxt.org/ 15. [S11] *"Why we created humans.txt"*, Creative Bloq (secondary, for the 2011 date and founders). https://www.creativebloq.com/netmag/why-we-created-humanstxt-11116628 **Primary — Google's position and tooling** 16. [S20] *"llms.txt"*, Lighthouse agentic browsing audit docs, Chrome for Developers — page updated 2026-05-05. https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt 17. [S27] *"Guide to Optimizing for Generative AI Features on Google Search"*, Google Search Central — updated 2026-07-10. https://developers.google.com/search/docs/fundamentals/ai-optimization-guide 18. [S24] John Mueller, Bluesky post — 2025-06-17 (permalink; not directly fetched). https://bsky.app/profile/johnmu.com/post/3lrshm4gggs2v 19. [S25] *"Google Says No AI System Currently Uses LLMs.txt"*, Search Engine Roundtable (**returned HTTP 403 to fetch**). https://www.seroundtable.com/google-ai-llms-txt-39607.html **Primary — standardisation** 20. [S35] W3C strategy issue #506, *"llms.txt"* — opened by xfq 2025-04-27, still open (Investigation). https://github.com/w3c/strategy/issues/506 21. [S37] IETF AI Preferences (aipref) working group drafts — vocab and attach. https://datatracker.ietf.org/doc/draft-ietf-aipref-vocab/ · https://datatracker.ietf.org/doc/draft-ietf-aipref-attach/ **Measurements and studies** 22. [S30] *"We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read"*, Louise Linehan with Xibeijia Guan, Ahrefs — 2026-06-15. https://ahrefs.com/blog/llmstxt-study/ 23. [S31] *"llms.txt and AI Visibility: Results from OtterlyAI's GEO Study"*, Otterly.ai (not directly fetched). https://otterly.ai/blog/the-llms-txt-experiment/ 24. [S33] *"LLMs.txt: Why Brands Rely On It and Why It Doesn't Work"*, SE Ranking — 2025-11-07, ~300k domains. https://seranking.com/blog/llms-txt/ 25. [S18] *"Does anyone actually have an llms.txt? I checked millions of websites"*, Casey Burridge — 2026-06-20, HTTP Archive/BigQuery. https://caseyrb.com/blog/state-of-llms-txt-adoption/ 26. [S38] *"LLMS.txt Adoption"* tracker, Rankability — Tranco top 1,000, monthly; page updated 2026-08-06. https://www.rankability.com/data/llms-txt-adoption/ 27. [S32] *"llms.txt adoption rises 8.8x but 97% of files get zero AI requests"*, Luis Rijo, PPC Land — 2026-07-02. https://ppc.land/llms-txt-adoption-rises-8-8x-but-97-of-files-get-zero-ai-requests/ **Commentary and criticism** 28. [S29] *"The llms.txt is dead. More precisely: a dud."*, Kai Spriestersbach, Medium — 2026-02-23. https://medium.com/@kaispriestersbach/the-llms-txt-is-dead-more-precisely-a-dud-ab7bee4f469c 29. [S26] *"Google Says llms.txt Is Purely Speculative For Now"*, Roger Montti, Search Engine Journal — 2026-06-02. https://www.searchenginejournal.com/google-says-llms-txt-is-purely-speculative-for-now/577576/ 30. [S19] *"Google's llms.txt Guidance Depends On Which Product You Ask"*, Matt G. Southern, Search Engine Journal — 2026-05-20. https://www.searchenginejournal.com/googles-llms-txt-guidance-depends-on-which-product-you-ask/575431/ 31. [S28] *"Google adds llms.txt check to Chrome Lighthouse"*, Danny Goodwin, Search Engine Land — 2026-05-20. https://searchengineland.com/google-llms-txt-chrome-lighthouse-478246 32. [S34] *"Meet llms.txt, a proposed standard for AI website content crawling"*, Rob Garner, Search Engine Land — 2025-03-28. https://searchengineland.com/llms-txt-proposed-standard-453676 **Directories** 33. [S21] llms.txt directory by @ifox and @joyceverheije. https://directory.llmstxt.cloud/ 34. [S22] llms.txt hub — 2,650 sites tracked at fetch time. https://llmstxthub.com/ 35. [S23] llmstxt.site — directory with token-size measurements. https://llmstxt.site/ **Low-confidence / flagged** 36. [S36] ⚠️ *"What The June 2026 W3C Proposal To Standardize llms.txt Means For Webflow Sites"*, Pravin Kumar — 2026-06-23. **Claims a W3C working draft that I could not verify and that appears not to exist. Cited here only so the article can flag it.** https://www.pravinkumar.co/blog/w3c-llms-txt-standard-proposal-june-2026-webflow-2026 37. [S17] Shopify llms.txt rollout coverage (secondary; no primary changelog located). https://craftshift.com/shopify-native-llms-txt-agentic-discovery-rollout/ · https://nikhil.pro/shopify-quietly-rolled-out-agentic-commerce-and-llms-txt-on-every-store 38. [S39] *"Shopify agents.md vs llms.txt: What Changed in 2026"*, Weaverse — dated 2026-05-28. https://weaverse.io/blogs/shopify-agents-md-llms-txt-theme-template-customization-may-28-2026 39. [S15] Secondary write-ups of the FastHTML origin anecdote. https://llms-txt.io/blog/is-llms-txt-dead · https://codersera.com/blog/llms-txt-complete-guide-2026/