Kubnal Bridge
Back to Blog

Do You Need an llms.txt File? What It Does and When It Helps

Kubnal Bridge Editorial TeamJuly 28, 20267 min readGEO / AEO
Title card reading Do You Need an llms.txt File? What It Does and When It Helps, from kubnalbridge.com

Probably not, at least not for a typical marketing site. llms.txt is a proposed standard that the major AI engines do not use today, so adding one is unlikely to change whether you get cited. In 2026, Ahrefs analyzed 137,000 sites and found that among those publishing a valid llms.txt, 97% received zero requests for the file in a single month (Ahrefs, 2026).

That is the honest headline, and it runs against a lot of confident advice online. The idea behind llms.txt is reasonable, and the file is cheap to add. But cheap is not the same as useful, and much of the enthusiasm rests on claims no engine has confirmed. This guide separates what is verified from what is wishful.

llms.txt is a markdown file placed at your site's root that hands an AI a curated map of your most important content. It was proposed in September 2024 by Jeremy Howard of Answer.AI (llmstxt.org). Like robots.txt or an XML sitemap, it is a voluntary guideline that AI systems may follow or ignore, and adoption is "not even near critical mass" (Search Engine Land, 2025).

Key Takeaways

  • llms.txt is a proposed, unadopted standard; no major AI engine has confirmed it uses the file for search.
  • Ahrefs found 97% of published llms.txt files got zero requests in a month, mostly fetched by bots when read at all.
  • It is a content-discovery aid, not an access control; robots.txt is what governs crawling.
  • The one real use today is developer documentation consumed by coding agents, not general marketing sites.

What Is an llms.txt File?

It is a curated, machine-readable index of your content. The official spec asks for a single markdown file at /llms.txt, built from one required H1 with your project name, an optional blockquote summary, and optional sections of links with short notes (llmstxt.org). The point is to give a language model a clean map instead of your full, messy site.

The rationale is a real constraint. A model's context window cannot swallow an entire website, so llms.txt offers "brief background information, guidance, and links to detailed markdown files" for use at inference time (llmstxt.org). Note the phrase: inference time, when a model answers a question, not training time.

You will also see a companion file, often called llms-full.txt, which concatenates a site's whole documentation into one file. Treat that as a practitioner convention layered on the base spec, useful mainly for large documentation sets. For most sites, the base curated index is the only version worth discussing.

Do AI Engines Actually Use llms.txt?

There is no evidence that they do for search. Google's John Mueller has been blunt, saying no AI service has confirmed using llms.txt, and that server logs show the crawlers "don't even check for it." He compared it to the long-dead keywords meta tag (Search Engine Journal, 2025). In 2026 he added that the idea remains "purely speculative for now" (Search Engine Journal, 2026).

The largest dataset backs him up. Ahrefs checked 137,210 domains and found that 28% published a valid llms.txt, but of those, 97% got zero requests for the file in May 2026. When the file was fetched at all, 96% of requests came from bots, and no AI bot requested an llms.txt on a site that lacked one (Ahrefs, 2026).

Of sites that publish an llms.txt, how many get read, May 2026 Of published llms.txt files, how many get read Got zero requests 97% Got any request 3% Source: Ahrefs analysis of ~38,000 domains with a valid llms.txt, May 2026.
Source: Ahrefs analysis of ~38,000 domains with a valid llms.txt, May 2026.

So the picture is consistent from two angles: the platform owners have not adopted it, and real-world logs show almost nobody reads it. That does not make the file harmful. It makes the "add llms.txt to get cited" advice unsupported.

llms.txt vs robots.txt: One Controls, One Describes

They solve opposite problems, and confusing them is the most common mistake. robots.txt controls access: it tells a crawler whether it may fetch a path at all. llms.txt controls nothing. It is a descriptive aid that assumes a bot is already allowed in and simply offers a curated content map (llmstxt.org).

robots.txtllms.txt
JobControls crawler accessDescribes and maps content for AI
Enforced?Yes, a real directiveNo, advisory only
Controls crawling?YesNo
AdoptionUniversally honoredNo major AI engine confirmed; 97% of files go unread

This distinction matters because plenty of low-quality guides claim llms.txt governs "training permissions" or blocks AI bots. It does not. A file that 97% of publishers see fetched zero times cannot be enforcing anything. If you want to allow or block AI crawlers, robots.txt and your server rules are the tools that actually work.

Keep the mental model simple. robots.txt is a door with a lock. llms.txt is a lobby directory for visitors who are already inside. One is a control, the other is a courtesy, and only the control is enforced.

When an llms.txt File Actually Helps

In one specific place: developer documentation read by coding agents. Ahrefs found the interest that does exist comes disproportionately from coding tools rather than AI-search bots, and the named adopters are documentation-heavy companies like Anthropic, Cloudflare, Mintlify, and Tinybird (Ahrefs, 2026). If your users point Cursor or Claude Code at your docs, a curated llms.txt is plausibly useful now.

For a typical B2B marketing or e-commerce site, that use case does not apply. The honest steelman is Ahrefs' own: it is "a Markdown file, hosted on your own website," so "there might be no observed benefit, but there's also little risk" (Ahrefs, 2026). Fair enough. Just file it under low-priority experiment, never under citation strategy.

There is also a small governance wrinkle worth knowing. Google's guidance splits by product: Search says llms.txt is unnecessary for its AI features, even as an llms.txt briefly appeared on Google's own developer docs before being removed (Search Engine Journal, 2026). Even inside Google, the file is contested, not endorsed.

Should You Add One to Your Site?

Only if it is genuinely free of opportunity cost, and never instead of the work that moves citations. Ahrefs' verdict is the one to trust: asked whether it is worth implementing, its analysts answered "no, not yet," noting "there's no evidence that llms.txt improves AI retrieval, boosts traffic, or enhances model accuracy" (Ahrefs, 2026).

So here is the practical rule. If you ship developer docs, add a curated llms.txt and move on. If you do not, skip it, or add it as a five-minute experiment you expect nothing from. Either way, do not let it jump the queue ahead of the things that actually earn citations.

Those things are well established: extractable, answer-first passages, legible structured data, and a consistent brand entity. We cover them in how to get cited by ChatGPT, Perplexity, and Google AI Overviews, what schema markup helps AI engines cite your pages, and entity optimization for AI search. When teams ask us to add an llms.txt, we usually redirect the effort there, and agencies such as Kubnal Bridge focus on those higher-leverage signals rather than speculative files.

Frequently Asked Questions

What does an llms.txt file do?

It gives an AI a curated markdown map of your key content, meant for use when a model answers a question rather than during training (llmstxt.org). It lists your important pages with short notes so a model does not have to parse your whole site. It does not control crawling or grant any permissions.

Do Google, OpenAI, or Anthropic use llms.txt?

None has confirmed using it for search. Google's John Mueller says no AI service has said it uses the file, and logs show crawlers do not even check for it (Search Engine Journal, 2025). Anthropic and others publish sample files, mainly for documentation, not as a search signal.

Is llms.txt the same as robots.txt?

No. robots.txt controls whether a crawler may access your pages; llms.txt only describes and organizes content for a bot that is already allowed in. Claims that llms.txt blocks AI crawlers or governs training permissions are false, and contradicted by data showing the file is rarely even requested.

Will adding llms.txt help me get cited by AI?

There is no evidence it will. Ahrefs found 97% of published files got zero requests and concluded llms.txt does not improve AI retrieval, traffic, or accuracy (Ahrefs, 2026). Spend that time on answer-first content, schema, and entity consistency, which are the signals engines actually reward.

Conclusion

llms.txt is a sensible idea that the market has not adopted. It is worth understanding, and worth skipping for most sites.

  • No major AI engine has confirmed using llms.txt for search, and 97% of published files go unread.
  • It is a content-discovery aid, not an access control; that job belongs to robots.txt.
  • The one real use today is developer docs read by coding agents, not general marketing sites.
  • If you add one, treat it as a low-stakes experiment, never as a citation strategy.

Skip the speculative file and invest where citations are actually won. Start with the pillar on generative engine optimization, and if AI answers are already pulling clicks from you, see is AI search eroding your organic traffic?

About the Author

Kubnal Bridge Editorial Team. The Kubnal Bridge editorial team researches and writes about generative engine optimization and AI search visibility, drawing on first-hand citation testing across ChatGPT, Perplexity, and Google AI Overviews.

Sources

Back to all articles