LLMs.txt 2026 In short, this means: The /llms.txt is a voluntary orientation file for AI systems, agents, and other automated readers. The file is a Community proposal, not an official web standard and not Access protectionCreating an LLMs file helps machines categorize your most important sources; however, it doesn't exclude anything.
This classification is important because around AI Discovery Files and AI discovery files often use a mix of different terms. The idea was explicitly published as a proposal on llmstxt.org on September 3, 2024. At the same time, robots.txt since RFC 9309 Although described as a Proposed Standard on the IETF Standards Track, according to the specification it does not constitute any form of authorization or access control.
From my work with SMEs, I've observed a recurring pattern: as soon as a new file format appears, people quickly look for a technical abbreviation. In practice, however, an LLMs.txt file is only beneficial if your website is already clearly structured, your services are comprehensibly named, and your core pages contain the official version of your LLMs.txt file. BrandDefinition of Brand: Brand (also called brands) is an English word for brand. A brand is a distinctive mark that identifies products or services... Click to learn more That's precisely why at Berger+Team we prioritize clarity, architecture, and substance first, and only then additional technical information.
The LLMs.txt file helps with understanding and prioritizing. LLMs.txt does not protect any content.
LLMs.txt 2026: Definition, status and limits
The LLMs.txt file is typically located in the root directory of your website under /llms.txt. There you link to the most important official sources of your domain, for example the homepage, service pages, FAQ, documentation, contact or other central content.
The status is clear for 2026: The LLMs.txt file will remain in use. no A formal standard from the IETF or W3C. The proposal describes a useful convention to help AI systems and assistants more quickly identify which content is relevant to a website. This is helpful for many companies, but you shouldn't mistake the file for a legally binding standard.
The original description on llmstxt.org explicitly calls the idea a proposal. This is crucial for its classification. The file can be useful, but its effectiveness depends on whether AI crawlers, tools, or agents actually read this convention and incorporate it into their processes.
What an LLMs file is useful for
A well-designed LLMs file reduces ambiguity. This is particularly valuable for owner-managed businesses, expert brands, and small teams, because these companies often don't have large content areas, but rather a few pages that need to be both technically and financially viable.
- Prioritization: You mark which URLs should be considered official primary sources.
- Context: You help systems distinguish between secondary content and core content.
- Citation ability: You direct machines to pages that are cleanly written, up-to-date, and brand-consistent.
- Orientation: You give assistants and tools a curated entry point instead of an unclear link landscape.
If you want to put the topic into a broader strategic context, our article on AI visibility for SMEsThis illustrates why technical files are just one component in a larger visibility system.
Clearly separate LLMs.txt, robots.txt and other files.
For SMEs, the distinction is usually more important than the file format itself. Once you know, which problem The correct file will become clearer once you want to solve it.
LLMs.txt
The LLMs.txt file serves as a guide to the content. It does not primarily specify who will teach something. crawlingCrawling means that a search engine bot like Googlebot or Bingbot automatically visits websites, follows links, and technically indexes their content. Without crawling, a page... Click to learn more It's not about what sources are allowed to be used, but rather which sources are important, official, and helpful in terms of content. Therefore, this file is particularly suitable if you want to prioritize information.
robots.txt
The robots.txt file controls crawl instructions for bots. The Robots Exclusion Protocol has been in effect since 2022. RFC 9309 Described as a Proposed Standard on the IETF Standards Track; colloquially, many refer to it as a IETF standardHowever, this classification is more precise. According to RFC 9309, these rules are explicitly no This form of access authorization is therefore not a substitute for protection mechanisms such as login, role rights or server-side blocks.
This isn't just theory. Anthropic documents several bots, including ClaudeBot, Claude-User, and Claude-SearchBot, and explains that website operators can control their access via robots.txt rules. This is precisely where you see the difference: robots.txt sends crawl signals, but it is not access control.
Robots Meta and X-Robots Tag
A Robots MetaThe `<h1>` tag operates at the HTML page level. X-Robots Day It works via HTTP headers and is therefore also useful for files or resources that are not directly HTML-based. Both mechanisms are more granular than a robots.txt file, but neither provides a complete barrier against direct access.
llms-full.txt
A file often appears in connection with LLMs.txt. llms-full.txt This usually refers to a more detailed accompanying file that contains significantly more content or full texts. Important for practical purposes: An llms-full.txt file is not automatically required and is not a mandatory part of the core proposal on llmstxt.org.
Agent Descriptor File and ai-plugin Manifest
A Agent Descriptor File or a AI plugin manifesto A system file describes the capabilities, interfaces, rules, or functional logic of a system. Such files are relevant when agents are to actively use tools, call APIs, or execute clearly defined actions. An LLMs.txt file, on the other hand, primarily describes orientation within the system. ContentContent encompasses all intentionally published digital content on websites, in online shops, on social media channels, in newsletters, and in other digital environments. If you want to know more... Click to learn more, not the executable functionality of a tool.
Simple decision logic for SMEs
- If you want to prioritize content: Use an LLMs.txt file.
- If you want to control crawling: Additionally, use a robots.txt file.
- If you want more granular control over the indexing of individual pages or files: Use Robots-Meta or X-Robots-Tag.
- If you provide features, tools, or agent capabilities: Use an Agent Descriptor File or an AI plugin manifest instead.
- If you really want to protect something: Use true access control, i.e., authentication, role rights, and server-side rules.
I'm deliberately phrasing this clearly because otherwise small businesses quickly waste time on the wrong problem. If PositioningAn ideal customer profile is a precise description of the company that best matches your offering, your working methods, and your business goals. A... Click to learn moreSince the offer structure and core pages are still unclear, no file will solve this problem.
Creating LLMs.txt: What will be considered best practice in 2026
If you want to create an LLMs.txt file, keep it small, official, and well-maintained. In most projects, a short, clean file is more effective than a long list without editorial control.
- List of a few official URLs: Homepage, core services, FAQ, contact, about us and important documentation.
- Use consistent brand terminology: The same service names, the same spelling, and the same responsibilities as on the website.
- Avoid internal or sensitive links: No preview pages, no staging environments, no protected documents.
- Clear ownership: Specify who releases the file and when it is updated.
- Check the core pages first: If the website's language is unclear, the LLMs.txt file will also be unclear.
It is precisely at this point that it is often worthwhile to first take a look at Machine-readable contentA file can only prioritize what is already clearly formulated on the linked pages.
What a minimal structure can look like
The proposal on llmstxt.org describes the basic idea as a Markdown file containing the project name, a brief description, and curated link lists. This minimal structure is often sufficient for SMEs:
- Website or brand name
- brief summary in one sentence
- A short list of the most important official URLs
- Optionally, a second list with supplementary, less central sources.
A pragmatic minimal example might look like this:
# Berger+Team
> Offizielle Informationen zu Leistungen, Beratung, Website und Kontakt.
## Wichtig
- https://www.berger.team/
- https://www.berger.team/leistungen/website/
- https://www.berger.team/leistungen/branding/
- https://www.berger.team/leistungen/beratung/
## Optional
- https://www.berger.team/ki-loesungen/The selection of formats is more important than the format itself. In my experience with small businesses, files become problematic when they become a dumping ground for everything that seems important internally at the moment.
Common errors in LLMs.txt
- Misunderstanding the file as a lock file: The LLMs.txt file is not a defense against AI crawlers.
- Including too many URLs: When everything is important, nothing is prioritized.
- List outdated pages: Machines can find content, but not the right kind.
- Publish without governance: Nobody feels responsible, the file is silently becoming outdated.
- Attempting to technically conceal strategic ambiguity: The unclear performance architecture remains unclear even with LLMs.txt.
What will have become established by 2026 and what won't.
The situation in 2026 is more sobering than many trend articles suggest. The understanding that websites, in addition to traditional methods, now play a crucial role has become firmly established. SEOSEO explained simply: SEO is the strategic optimization of a website for organic visibility, relevant search intent, and clear presentation in search engines and search-related response systems. The goal... Click to learn more They also need to be better machine-readable, citable, and logically structured. This includes clear page hierarchies, good FAQs, unambiguous service pages, and a consistent brand.
The idea that a single file automatically creates visibility, control, or protection has not become established. The LLMs.txt file remains a useful convention. For some websites, it makes sense; for others, the website architecture is the more significant factor. You can find a broader overview of related file types in our article on... Key files for AI-powered websites.
When the LLMs.txt is really worthwhile for SMEs
An LLMs.txt file is particularly worthwhile if your website is already a clear primary source and you want to further highlight this quality. This often applies to consultancies, specialized service providers, software products, knowledge bases, and companies with well-maintained FAQs or documentation.
The file is less urgent if you're still struggling with fundamental issues: unclear service pages, no clear positioning, scattered contact persons, contradictory terminology, or a missing FAQ. In such cases, I almost always invest in structure and brand logic with clients first. Otherwise, a well-intentioned LLMs file will just become another technical document on an already messy website.
Frequently Asked Questions about LLMs.txt
Is LLMs.txt an official standard?
No. The LLMs.txt file is a community proposal and not a formal web standard from the IETF or W3C. Therefore, you should understand the file as a helpful convention, not as a binding technical standard.
Does LLMs.txt replace robots.txt?
No. The LLMs.txt file complements the robots.txt file because both files have different functions. The LLMs.txt file prioritizes content, while the robots.txt file provides crawl instructions for bots.
Can I block AI crawlers using the LLMs.txt file?
No. If you want to control AI crawlers or other bots, you'll need robots.txt rules and, depending on the situation, other technical measures. If you truly want to protect content, you need real access control, not a public text file.
Do I also need an llms-full.txt file?
Not automatically. An llms-full.txt file can be useful if you intentionally want to provide a more detailed accompanying file with more context. However, for most SMEs, a lean LLMs.txt file with clearly prioritized core sources is sufficient to begin with.
Should the LLMs.txt file point to the sitemap?
This can be useful if the sitemap is a helpful addition. However, the sitemap does not replace the LLMs.txt file. The sitemap is usually comprehensive, while the LLMs.txt file should be carefully curated and prioritized.
Can I list multiple language versions?
Yes, provided the language versions are officially maintained and clearly labelled. This is particularly useful for South Tyrolean or internationally operating SMEs, as long as German, Italian, and English are clearly separated and consistently named.
How often should I update the LLMs.txt file?
This should be done whenever core pages, services, contact persons, or important FAQs change. For many small businesses, a regular quarterly check is also sufficient to ensure no outdated URLs remain active.
What should not be included in an LLMs file?
Internal links, sensitive documents, unfinished pages, or anything not suitable as an official source should not be included. A good LLMs file is curated, not exhaustive.
Conclusion
The LLMs.txt file is a useful, but clearly limited, orientation file in 2026. It helps machines categorize your most important sources, but it doesn't replace anything. robots.txt or Robots Meta, X-Robots Day, Agent Descriptor File, AI plugin manifesto or real Access protection.
My practical conclusion from over 20 years of working with small businesses is simple: For SMEs, the file is only useful if their positioning, services, core pages, and FAQs are already clearly defined. Clarity comes first, then the file. Everything else is just technology without a strategic foundation.
