The llms.txt is a useful tool today, primarily for IDE agents and developer documentation. For mainstream chat-bots like ChatGPT or Google AI, broad, consistent adoption is currently not demonstrable.
- The
llms.txtis a Markdown file in the root directory that shows AI systems your most important content, especially relevant for IDE agents like Cursor. - Anthropic, OpenAI and Perplexity use
llms.txtfor their own documentation, but no provider has declared it a standard for AI answers. When SE Ranking surveyed nearly 300,000 domains in November 2025, around 90% had none. - GPTBot crawls llms.txt inconsistently. Coding agents like Cursor or Cline read it when you hand it to them as context. This is the strategically more important use case today.
- Implementation takes under 30 minutes, whether manually, via Yoast SEO or plugin. Despite mixed study results, it is worth doing now.
Correction of 18 September 2026: Two further statements in this guide were not supported and have been replaced. First, it said that in some cases “removing the file even had a neutral to slightly positive effect on model accuracy”. That was a misreading of the SE Ranking study: what was removed there was the llms.txt variable from the company’s own prediction model, whose accuracy improved as a result. It says nothing about websites that delete the file. Second, it said of my log screenshot that it was “the decisive proof” that the infrastructure for processing the file is in active use. A request in a log documents a request, not processing. Sources: SE Ranking, 7 November 2025; John Mueller, Bluesky, 17 September 2026.
As a Product Developer in the iGaming sector and SEO freelancer, I live at the intersection of user behavior and technology. For me, the revolution of web search is not a theory. It decides success or failure on a daily basis. And recently I saw it in black and white in my own server logs: the AIs are knocking.
Imagine being able to whisper directly into the ear of an AI like ChatGPT or Google AI Overviews, telling it exactly how to talk about your brand. That possibility is becoming reality right now.
We are facing the biggest upheaval since the invention of Google search. The hunt for “blue links” is giving way to a new reality: Generative AIs serve up ready-made answers on a silver platter. This turns classic SEO on its head and calls for new disciplines like Generative Engine Optimization (GEO).
And this is exactly where the llms.txt enters the stage. Where it demonstrably works today and where it does not can now be answered with numbers rather than guesses. That is what this post does.
Ready to separate the hype from reality? Perfect! In this post, I’ll give you a clear, actionable roadmap, practical, honest, and based on solid data.
What exactly is an llms.txt file?
An llms.txt is a simple text file written in Markdown format, placed in the root directory of your website to provide language models (LLMs) with a curated list of your most important content. It is optimized for both humans and machines and serves as a guide to the content that best defines your expertise and brand identity.
The anatomy of llms.txt: The official standard
The official specification on llmstxt.org has been at version 2 since 10 August 2026. The structure of the file itself has not changed:
- H1 heading (mandatory): The file always starts with an H1 (
#) containing the name of your website. - Blockquote summary (optional): Directly below, a blockquote (
>) can summarize your site briefly and concisely. - H2 sections as “file lists” (optional): H2 headings (
##) let you group links to your most important content. - Link list syntax: Each link is written in Markdown format
[Link-Name](URL), optionally followed by a short description.
New in v2 (August 2026): The proposal now describes how an agent finds the file in the first place. A <link rel="describedby"> in the HTML or an HTTP Link: header points to the llms.txt that covers the page, and a <link rel="alternate" type="text/markdown"> points to the page’s Markdown version. Both URL forms are allowed for that Markdown version, page.html.md and page.md. An llms.txt in a subpath covers the pages under that path, and the most specific file wins. The earlier tooling for context expansion is no longer part of the proposal; the “Optional” section is kept as a convention but no longer carries technical meaning (llmstxt.org, Changes v2).
The crucial difference: llms.txt vs. llms-full.txt
There are two variants, and this distinction is crucial for your strategy:
llms.txt: Think of it as a table of contents or a map. It contains only links to your most important content. The AI has to follow these links to get to the details.llms-full.txt: This is the compendium, the complete text. It bundles the entire content of your most important documents into one large file. This saves the AI from subsequent crawling and is pure gold for RAG systems (Retrieval-Augmented Generation).llms-ctx.txt: An expanded variant without optional URLs, proposed in 2025 and generated with thellms_txt2ctxtool. Since v2 of the proposal (August 2026) this tooling is no longer part of it; agents are expected to read the llms.txt and follow the links rather than expanding it in advance.
As a product developer, I immediately see the value of llms-full.txt. Imagine having a complex knowledge base for an iGaming product. Handing an AI that information as a single, clean file instead of sending it through dozens of HTML pages is a massive efficiency gain. This is not theory. It is a direct response to the architecture of modern AI systems.
The core problem it solves: Why HTML is “noisy” for LLMs
Why the effort? Because a modern webpage is a genuine challenge for an AI. HTML is full of “noise” for an AI, navigation, ads, and scripts that distract from the core information.
- Context window limitation: An AI has a limited attention span (the “context window”). This HTML noise consumes valuable space.
- Faulty tokenization: AIs break text into “tokens”. Complex code can disrupt this process and lead to misinterpretations.
- Inefficiency and cost: Parsing HTML is computationally intensive and expensive for an AI. Markdown, on the other hand, is essentially its native language, clean, efficient, and to the point.
What log files show: requests yes, broad usage no
Log file analysis shows: certain crawlers, including occasional visits from OpenAI’s GPTBot, request the /llms.txt file. However, studies show no consistent, broad adoption by the major chat-bot systems. Here is an excerpt from my own log files as a real-world example:
You can see bots checking whether llms.txt exists. The screenshot documents nothing beyond the request. SEO expert Ray Martinez reported in July 2025 that OpenAI fetched the file on some of his sites.
John Mueller has since explained where such a request can come from. There are directory sites that check every domain for an llms.txt and link to it. When a crawler sees such a page, it follows the links, and the request shows up in the log. In his words: “There are a bunch of “everything llms.txt” sites out there now that try to check all domains for the llms.txt file and link to it from their site. So when a crawler sees their site, they see the links, and follow them.” (John Mueller, Bluesky, 17 September 2026). Such measurements, he added, are only meaningful “at the moment”. llmstxt.org itself lists three of these directories: llmstxt.site, directory.llmstxt.cloud and llmstxthub.com. A hit in the log is therefore not proof of usage, but first of all a link that someone followed.
Since 17 September there is a measurement that picks up exactly here. Nathan Hall logged seven WordPress sites on one server for 31 days and asked a different question than the studies before it. Not whether crawlers find a file nobody tells them about. But whether they fetch one whose address the site itself names. In robots.txt via a Sitemap line and in the HTML head via a link tag. In that setup no directory site can explain the requests. Across 165 site-days the announced file was fetched 1,549 times, 9.4 per site per day on average. Split by provider, the picture falls apart immediately.
| Crawler | llms.txt fetches | Requests in total |
|---|---|---|
| ClaudeBot (Anthropic) | 1,131 | 3,599 |
| Bingbot (Microsoft) | 310 | 1,047 |
| GPTBot (OpenAI) | 106 | 826 |
| Googlebot (Google) | 0 | 939 |
Googlebot made 939 requests in those 31 days and not one of them was for an llms.txt. ClaudeBot fetched the file a median eight seconds after the robots.txt of the same host. It fetched the XML sitemap named in that same robots.txt just as often. And that is exactly why the number says nothing about usage: tell a crawler an address and it fetches it. Hall put that in his title: Discovery, not adoption. The abstract spells it out: “They do not establish recognition of llms.txt, parsing, indexing, ranking, or use in AI-generated answers.” (Nathan Hall, Zenodo, 17 September 2026)
A second finding explains why older counts come out so low. The un-announced file in the root directory sat on all seven sites. Verified crawlers fetched it exactly twice in 31 days. It was requested 567 times. The other 565 requests came from unidentified user agents, SEO tooling and scanners. Watch only /llms.txt and you mostly measure our own industry.
These log file data show: the infrastructure to process llms.txt exists, but chat-bots are barely using it yet. An honest update: a study by SE Ranking (7 November 2025) analyzing nearly 300,000 domains found no relationship between an llms.txt and how often a domain is cited in AI answers. When SE Ranking removed the variable from its own prediction model, the model even became more accurate; for the model, the file was noise rather than signal. SE Ranking limits its own finding here. It holds only for the model and the dataset tested, in their words “context-dependent”. This is no reason to abandon implementation, but a clear argument for keeping expectations realistic. Because the biggest benefit today lies elsewhere.
llms.txt vs. robots.txt vs. sitemap.xml: What’s the difference?
The main difference lies in purpose and audience: robots.txt issues access restrictions to crawlers, sitemap.xml lists all URLs for discovery, and llms.txt provides qualitative guidance for AI systems.
The easiest way to understand it is with this analogy:
robots.txtis the security guard of your website. It tells crawlers which areas are off-limits. A pure access control protocol.sitemap.xmlis the floor plan of your building. It lists all rooms (pages) so traditional search engines can find everything.llms.txtis the personal VIP tour guide for the AI. It leads the LLM directly to the highlights of your website and explains why they matter.
The three musketeers of website control
| Criterion | llms.txt | robots.txt | sitemap.xml |
|---|---|---|---|
| Main purpose | Guidance & context | Access control | Discovery & indexing |
| Audience | LLMs, AI agents | Search engine crawlers | Search engine crawlers |
| Format | Markdown | Plain Text | XML |
| Strategic focus | GEO, narrative control | Technical SEO (exclusion) | Technical SEO (inclusion) |
Why is llms.txt so hotly debated?
The debate ignited over the initial lack of official adoption, but has shifted from a theoretical to a practical discussion as log file evidence emerged.
The criticism: missing adoption and legitimate skepticism
The skeptics’ arguments were long compelling:
- No official confirmation: For a long time no major player officially announced adoption, this has partially changed (more on that below).
- The “Keywords Meta Tag” analogy: Google’s John Mueller compared
llms.txtto the obsolete Keywords tag. As he is quoted in the Search Engine Roundtable, a trustworthy AI should verify a website’s claims regardless. - SERanking study November 2025: An analysis of over 300,000 domains found no measurable improvement in AI citations. In some cases, removing the file had a neutral to slightly positive effect on model accuracy.
- High maintenance effort: The benefit for classic search remains speculative.
Industry adoption: developer tools and documentation
Despite the mixed study results, the industry has set a clear course: Anthropic introduced llms.txt in November 2024 for its entire documentation, in collaboration with the documentation tool Mintlify. Overnight, thousands of hosted documentation pages followed the standard. OpenAI and Perplexity also have their own llms.txt files. And even Google serves its own llms.txt for the Gemini API documentation, and Chrome’s Lighthouse has checked since version 13.3 (7 May 2026) in its “Agentic browsing” category whether a site has the file; if it is missing, the check counts as not applicable, and only a server error is flagged (Chrome for Developers). Since version 13.5 of 18 September 2026 that check sits together with a new Agent Resource Discovery audit under the “agent discovery” group. For Google Search, Mueller’s assessment stands that no AI system uses the file for answers. Google’s own documentation is the clearest source on this: “You don’t need to create new machine readable files, AI text files, or markup to appear in these features” (Search Central, last updated 10 Dec 2025). This shows: the file has its place, but a different one than originally hoped.
For me as an SEO in the competitive iGaming industry, this is a clear signal. The question is no longer whether, but in which context it helps you today: for IDE agents, developer documentation, and structured AI communication. For chat-bot rankings and citations, you should keep expectations low. When SE Ranking surveyed nearly 300,000 domains in November 2025, around 90% had no llms.txt. Whether that becomes an advantage depends on whether the systems that do fetch it are the ones you care about.
How do I create an llms.txt? A step-by-step guide
You can create an llms.txt file either manually with a text editor for maximum control, or conveniently via plugins like Yoast SEO for WordPress.
Method 1: Manual creation: The strategic gold standard
- Create a new text file: Open a simple text editor (like Notepad on Windows or TextEdit on Mac) and name the file
llms.txt. - Copy and adapt the template: Paste the following text into your file and replace the placeholders
[...]with your own website’s information.
# [Your company or project name] > This is a summary of my website for AI models like Gemini or ChatGPT. > Contact: [Your email address] > Last updated: [Today's date] About me I am [a short, clear description of your work]. My mission: [Link to your "About" page] Get in touch: [Link to your contact page] My most important content [Title of your most important offer or blog post] - [Link to the corresponding page] [Title of another important piece of content] - [Link to the corresponding page] Follow me LinkedIn: [Link to your LinkedIn profile]
- Upload the file: Upload the finished
llms.txtfile to the root directory of your website. This is the same location as yourrobots.txt.
Done! You have now given the AI a clear guide. Verify the file by visiting yourdomain.com/llms.txt in your browser.
Method 2: Yoast SEO integration (The easy start for WordPress)
- Navigate to
Yoast SEO > Settings > Site features. - Scroll down to the
APIssection. - Enable the toggle for
llms.txt fileand save.
Yoast will now automatically generate a basic llms.txt. Learn more directly at Yoast SEO. Note: Automation is convenient, but rarely strategically optimal.
Method 3: Alternatives for all systems (generators)
- For WordPress: The plugin “LLMs.txt and LLMs-Full.txt Generator” is a solid alternative with more configuration options.
- For all systems: Online generators like Firecrawl can create a first draft. Important: Use these tools as a starting point, but always review and refine the result manually!
What does an optimal llms.txt look like in practice?
An optimal llms.txt is a manually curated selection of content that strategically reflects the core competency of the website, rather than just automatically listing all pages.
Here at seo-kreativ.de, I currently use the plugin mentioned above. You can view the llms.txt of seo-kreativ.de live. For strategic fine-tuning, I recommend manually selecting and describing the most important “crown jewels” of your website.
How does llms.txt fit into a comprehensive GEO strategy?
The llms.txt is a key building block of Generative Engine Optimization (GEO), directing AIs straight to content already optimized for machine processing. It is the signpost, but the content itself must be the destination.
GEO is the optimization of your content so that an AI can not only find it, but effectively use it for an answer. The focus is on:
- Clarity and precise answers
- Structure through headings and lists
- Trustworthiness (E-E-A-T)
- “Portability” of content snippets (information units that remain correct even out of context).
In my SEO work, E-E-A-T (Experience, Expertise, Authoritativeness, Trustworthiness) is absolutely critical. What many don’t realize: these signals matter not just for humans. An AI deciding which source to trust also looks for these signals. GEO means making E-E-A-T machine-readable, through clear author information, structured data, and references to authoritative sources.
Further key GEO levers
- Semantic structure: Use short paragraphs, clear headings, lists, and question-and-answer formats.
- Structured data (Schema.org): Use markup like
FAQPage,HowToorArticle. - Multimodal optimization: Provide high-quality, unique, and well-described (alt texts!) visual media.
FAQ: Your most pressing questions about llms.txt
Does an llms.txt file harm my traditional SEO?
No, absolutely not. The file targets only LLMs and is ignored by traditional search engine crawlers like Googlebot. One 31-day log file measurement counted 939 Googlebot requests and not a single one for an llms.txt. There are no negative effects on your rankings.
Do I really need to convert my pages to Markdown?
It is the recommended best practice, as Markdown is the cleanest format for AIs to process. If that’s not technically feasible, a link to a clean HTML page is still better than no entry at all.
Do Google, ChatGPT & Co. officially use the file now?
No, not for mainstream chat-bots. Google’s John Mueller wrote in June 2025: “FWIW no AI system currently uses llms.txt.” In September 2026 he added that requests for the file often come only from directory sites linking to it. Anthropic, OpenAI and Perplexity have llms.txt files for their own documentation, but no provider has declared the file a central feature for AI answers. The clearest demonstrable benefit today: coding agents like Cursor read llms.txt for developer documentation when you hand it to them as context.
Can I use llms.txt to prevent my data from being used for AI training?
No, that’s not what it’s for. Training is controlled via robots.txt (e.g., with User-agent: GPTBot and Disallow: /). llms.txt guides the AI when answering a specific query, not during general training.
How many links should I include in my llms.txt?
Quality over quantity. Focus on 5-10 strategically selected links that represent the heart of your website, your core competencies, and your most trustworthy content.
What happens if I don’t keep my llms.txt up to date?
That’s a real risk. An outdated file can cause an AI to use incorrect or outdated information, which can harm your brand and your E-E-A-T. Regular maintenance is essential!
Does llms.txt improve my citations in ChatGPT or Google AI?
Not demonstrably. The two pieces of evidence are of a different kind though. SE Ranking measured it in November 2025 across nearly 300,000 domains: no relationship between an llms.txt and how often a domain is cited in AI answers. Which AI systems were in that sample, the study does not say. For Google the statement comes from Google itself. John Mueller: “FWIW no AI system currently uses llms.txt.” Implementation is worthwhile nonetheless: as an entry point for coding agents (Cursor, Cline) and for future adoption, but not as a reliable tool for more chat-bot citations.
Should I set my llms.txt to noindex?
Yes, that makes a lot of sense. The llms.txt file is an instruction for AI systems, not content for users in search results. A noindex tag prevents it from appearing in the normal Google index. AI crawlers can still find and use the file for their purpose.
Conclusion: Your roadmap for the AI-optimized website
That was a lot to take in! But hopefully things are clearer now. The llms.txt is no silver bullet, but a sensible tool with a clearly defined scope: developer documentation, IDE agents, structured AI communication. For chat-bot rankings, you should not expect miracles; the data says so clearly.
Broader adoption may come as the infrastructure matures. Until then: llms.txt is a sensible, low-cost step, both for the developer community today and as early positioning for possible broader AI adoption tomorrow.
