The 'llms.txt' File: Should Your Site Have One in 2026?

Anthropic, Stripe, Cursor, Cloudflare, Vercel, Mintlify, Supabase, and LangGraph all ship an llms.txt file at the root of their docs domains. Around 10% of websites overall now have one. But the inconvenient truth is that no major AI crawler — GPTBot, ClaudeBot, PerplexityBot, Google-Extended — officially requests llms.txt at any meaningful volume. Server-log audits show GPTBot occasionally fetches it; the rest essentially do not. So: should you ship one? Short answer: yes, with caveats and a clear understanding of what it does and does not do. This post is the honest take, the copy-paste template we use for Softechinfra-tier services businesses, and the trade-offs. ## TL;DR — should you ship it Yes if you are a docs-heavy domain, a SaaS, or a services firm with 30+ valuable URLs. No if your site is 8 pages and a contact form. The cost is two hours and 5 KB. The upside is being on the right side of a future standard if Anthropic or OpenAI ever decide to honor it; the downside is essentially zero. Treat llms.txt as an entity-signal investment and a documentation discipline exercise — not as a magic citation lever. The current 2026 reality is what Google's John Mueller said on Reddit: AI engines do not need a separate file to read your site, they can read it directly. ## What is llms.txt, in 30 seconds llms.txt was proposed by Jeremy Howard (Answer.AI, fast.ai) in September 2024 as a Markdown-formatted file at the root of a website — /llms.txt — that tells large language models which content on the site is most important to read and in what order. The spec is intentionally simple: a Markdown document with an H1 title, an optional short description, and a list of grouped Markdown links to your most useful pages. A common companion file, llms-full.txt, contains the actual Markdown bodies of those pages so an LLM can ingest the entire useful corpus in one fetch without spidering the HTML. ## Why this matters now (April 2026 trigger) Three things happened in Q1 2026 that put llms.txt back on every SEO meeting agenda. First, the Discoverability Co schema study showed pages with comprehensive structured data lift AI citations by 44% — and llms.txt is structured data's lazy cousin. Second, the Mintlify "real llms.txt examples" roundup went viral on r/programming. Third, Google's John Mueller publicly likened llms.txt to the old "keywords" meta tag — which the SEO community read as "Google does not care, but it costs nothing to ship." The split between "ship it for safety" and "do not waste time on it" defines the 2026 debate. ## Which AI labs and tools actually read it (the honest list) The honest pattern: if your llms.txt is going to be read at all today, it will be by AI coding assistants and Claude-family tooling on docs-heavy domains. For an Indian B2B services firm, the immediate utility is small. The medium-term bet is whether a standard emerges. ## When to ship llms.txt (the decision matrix) | Site type | Ship llms.txt? | Why | |---|---|---| | Docs site / API reference | Yes, today | AI coding tools fetch it. Direct user value. | | Services firm with 50+ blog posts | Yes, this quarter | Entity-graph signal, free, sets discipline | | 8-page brochure site | No | Not enough content to need a curated index | | Edtech / consumer SaaS | Maybe | Useful if your product is dev-adjacent or has public docs | | E-commerce | Probably not | llms.txt is text-content focused; not designed for catalog | | Affiliate / SEO content farm | No | The signal you broadcast is "we want LLMs to scrape us harder," which is the opposite of what you actually want | ## A copy-paste llms.txt for a Softechinfra-tier services firm This is what we ship for our own site and for the 14-person Pune logistics-tech client we wrote up in another post. Adapt for your domain. Three things to notice. First, the description block (the > line) is the most-cited line — keep it tight, name your entity, name what you do, name where you operate. Second, the link list uses Markdown anchor format with a short description per link — this is the spec, follow it. Third, group your links by purpose (Services / Founder / Products / Blog / Contact) so any LLM ingesting it has clear sections to reason over. ## llms-full.txt — when to ship the longer variant The companion file llms-full.txt contains the full Markdown body of the pages listed in llms.txt. Anthropic, Vercel, and LangGraph all ship both. The use case is: an AI coding tool wants to ingest your entire useful documentation corpus in one fetch instead of crawling 30 HTML pages. For a services firm, llms-full.txt makes sense only if your blog is genuinely useful as a knowledge corpus to an LLM. If your blog is mostly news-recap and "10 reasons to use Cloud," skip it. If your blog has technical depth — actual code, real numbers, named clients — ship llms-full.txt with your top 20–30 posts concatenated as Markdown. ## The DIY ship plan (2 hours, including validation) Quick pre-flight checklist:

• You can write to /public/llms.txt or your hosting equivalent
• You have a list of your 20 to 30 most valuable URLs ready
• You have a tight 1-line description of who you are, what you do, who you serve
• You have decided whether to ship llms-full.txt as well (yes if your blog is technical and useful)
• You have stripped any client names or NDA-covered details from the corpus you plan to publish

## The r/SEO debate — the honest read If you spend an hour on r/SEO and r/bigseo threads from Q1 2026, the consensus is mixed. The skeptical camp (loudest) says: AI labs do not request it, John Mueller compared it to the keywords meta tag, it is a waste of time. The pragmatic camp says: it costs 2 hours, the file is 5 KB, and there is no downside. The bullish camp says: the standard could be formalized in 2027 and you want to be early. Our take after shipping it for ~40 client sites since mid-2025: the direct citation impact is unmeasurable. The indirect impact — forcing a team to write a clear, structured description of what they do and what their best content is — is significant. It is a documentation discipline exercise dressed up as an SEO tactic. If you have docs-heavy infrastructure, ship it now. If you do not, ship it when your blog hits 30+ valuable posts. ## Common mistakes Treating llms.txt as a sitemap.xml replacement. They are different. sitemap.xml lists every URL for indexing crawlers. llms.txt curates the most useful URLs for LLM context. Ship both. Stuffing every URL into llms.txt. The spec is about curation. A site that lists 800 URLs in llms.txt is signaling "I do not know what is important on my own site," which is the opposite of useful. Forgetting to update it. llms.txt is a living document. When you publish a major new post, add it. When a service page deprecates, remove it. We rebuild ours quarterly. Mismatched URLs. Every URL in llms.txt must return 200. Broken links signal a low-quality site to any LLM that does fetch the file. No description block. The > quote line is the part most likely to be lifted into AI answers. Skip it and you waste the most-cited line in the file. ## A real example We shipped llms.txt + llms-full.txt for a 6-person Mumbai fintech client in February 2026. They had 22 blog posts and an active developer audience. After 8 weeks, server logs showed Claude Code and Cursor fetching their llms.txt regularly when their users asked AI tools about Indian KYC and UPI integration. Did this translate to leads? Indirectly — three engineering leads who later became clients said they "knew the brand from being recommended by Claude." Not direct attribution, but consistent enough to repeat the exercise for every dev-tool client. For the founder's perspective on why standards like llms.txt matter even when adoption is slow, our founder Vivek Singh writes about early-standard bets on his blog. ## FAQ ### Does Google read llms.txt? No, not in any meaningful volume as of May 2026. Google-Extended does not fetch it. Google's John Mueller publicly compared the file to the old "keywords" meta tag, suggesting Google has no plans to use it. ### Does Anthropic's ClaudeBot read llms.txt? Server-log audits suggest no, not at scale. But Anthropic's own user-facing AI tooling (Claude Code, Claude Skills) does fetch llms.txt for context on docs domains. So Anthropic the company benefits from your file even if ClaudeBot the crawler does not log many requests. ### Can I put llms.txt in a subdirectory like /docs/llms.txt? You can, but the convention is the root domain. If you have a docs subdomain (docs.yoursite.com), put it at docs.yoursite.com/llms.txt. Most readers expect root placement. ### Does llms.txt replace robots.txt? No. They solve different problems. robots.txt tells crawlers what they can and cannot fetch. llms.txt tells LLMs what is most important on your site. Ship both, and use robots.txt to allow AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) if you want to be cited. ### Should I block AI crawlers in robots.txt and skip llms.txt entirely? Only if your business model genuinely depends on traffic that is dying — affiliate sites where the user must click through to monetize. For every other model, allowing AI crawlers and shipping llms.txt is the dominant strategy. Blocking removes you from citation pools, which is the new top of funnel. ### What is the file size limit? There is no hard limit, but Anthropic's llms-full.txt is ~2 MB and tools generally handle that fine. We keep llms.txt under 10 KB and llms-full.txt under 5 MB. ### Will llms.txt still matter in 2027? Honest answer: nobody knows. Either it becomes a standard with Anthropic or OpenAI signing on publicly, or it joins the meta-keywords graveyard. The 2-hour investment hedges both outcomes.