If you know the difference between robots.txt and meta robots If you want to understand, a clear separation is sufficient: robots.txt controls the Crawling, Meta-Robots and X-Robots Day support those Indexing and partly the representation in the search engine, and llms.txt is at most voluntary help for AI DiscoveryThese three levels are often mixed up in SME projects. The result is pages that are blocked but still appear, or pages with... noindex, which Google can never read.
I've seen this for years in website relaunches, e-commerce platforms, and multilingual company websites: the technology is often not the real problem. The problem is that one signal is mistaken for accessibility, another for visibility, and a third for AI orientation. If you clearly separate these, you avoid typical SEO mistakes, unnecessary ambiguity in the index, and a lot of time spent troubleshooting.
Mnemonic: robots.txt tells a search engine what it is allowed to crawl. Meta-Robots tells a search engine what should happen to a specific URL in its index. llms.txt voluntarily describes content for AI systems, but is not a legally binding access control measure.
robots.txt and Meta-Robots: What is the difference?
The comparison between robots.txt and meta robots This is fundamental work. Both signals operate at different points, have different ranges, and solve different tasks.
robots.txt: Purpose, location, effect, boundaries
- Purpose: robots.txt controls the crawling. You give bots instructions on which areas they should or should not access.
- Location of implementation: The file is located in the root directory of a domain. It works with information such as... User agent and Disallow.
- Effect: The file can restrict access to directories, file types, or URL patterns for specific crawlers.
- Limits: robots.txt is no A reliable method to remove a URL from the Google index. However, a blocked URL may still remain known and appear in search results.
Meta-Robots and X-Robots tag: Purpose, location, effect, limits
- Purpose: Meta-Robots It primarily controls the indexing of individual pages. Typical settings include: noindex or nofollow.
- Location of implementation: The meta robots tag is located in the HTML head of a specific page. X-Robots Day In contrast, it is set in the HTTP header and is therefore also practical for PDFs, images or other non-HTML files.
- Effect: This tells a search engine whether a URL should be indexed and how to handle signals from that URL.
- Limits: The search engine must be able to retrieve the resource in order to even see the meta robots tag or X robots tag.
llms.txt: Purpose, location, effect, limits
- Purpose: llms.txt is intended to provide AI systems with a curated overview of important content on a website.
- Location of implementation: Usually located in the root directory of the domain as a separate file.
- Effect: llms.txt can the AI Discovery to support, i.e., finding and classifying relevant content for LLM-based systems.
- Limits: llms.txt is not a recognized web standard, not a replacement for robots.txt, and not a robust mechanism for access control or indexing control.
The most important misconceptions briefly explained
1. A URL blocked via robots.txt is not automatically removed from Google.
This is one of the most common misconceptions. Google explicitly documents that a URL can appear in search results despite robots.txt blocking if Google knows the URL through other signals, for example, internal or external links: Source: Google Search CentralThe robots.txt file prevents the crawling of the content, but not necessarily the knowledge of the URL.
2. noindex only works if Google is allowed to read the page.
This is also often implemented incorrectly. According to Google, Googlebot must be able to crawl a page to execute a meta robots tag. noindex , or X-Robots Day to be visible at all; if the URL is blocked via robots.txt, this rule will not be read and will therefore be ignored: Source: Google Search Central. In short: Disallow plus noindex In many cases, this is a bad combination if you really want to remove a URL from the index.
3. llms.txt is not a replacement for robots.txt
There's a lot of attention currently surrounding llms.txt, but it's important to keep the classification clear. The specification is developed by Jeremy Howard and Answer.AI themselves as Proposal and an informal specification described with community input, not as an officially recognized web standard: Source: GitHubTherefore, llms.txt is more of a voluntary guideline for AI systems than a technical control instrument.
If you want X, use Y.
- You want to limit crawling in certain areas: use robots.txt with clearly defined User agent and Disallow.
- You don't want a specific page to be in the Google index: use Meta robots with noindex , or X-Robots Day and allow crawling until Google has processed the signal.
- You want to control PDFs or other files that don't have an HTML head: Use that X-Robots Day.
- You want to provide AI systems with guidance on important content: use llms.txt as a voluntary addition, not as access protection.
- You really want to protect sensitive content: Use authentication, rights management, or remove the content entirely. Neither robots.txt nor llms.txt are intended for this purpose.
Typical misconfigurations on SME websites
Especially in owner-managed businesses and small teams, I repeatedly see the same patterns in audits. These usually arise not from negligence, but from time pressure, plugin default settings, or a relaunch without clear technical responsibility. That's precisely why we examine these issues in our... Web design & development not only layout and content, but also the signal logic for SEO, indexing and machine-readable accessibility.
- noindex and Disallow simultaneously: The page is supposed to disappear, but at the same time it's blocked for Googlebot. As a result, Google often can't read the noindex signal.
- Block staging or test pages only via robots.txt: This isn't real protection. Password protection is much more reliable for preview or customer areas.
- Global nofollow settings without a strategy: This is sometimes activated due to uncertainty, without it being clear which internal signals are thereby weakened.
- llms.txt as a supposed control over AI access: An llms.txt file can provide guidance, but it is not a binding mechanism to technically block bots.
- Plugin configuration without overall view: A search engine does not just evaluate a single tag, but the interplay of crawling, status codes, internal linking, canonicals and index signals.
Practical logic for SMEs: Fewer files, more clarity
My recommendation is almost always the same: Use each file only for its intended purpose. A good website doesn't need excessive technical control, but rather clear responsibilities. Use robots.txt for crawl rules, meta robots or the X-Robots tag for indexing rules, and llms.txt for optional AI guidance.
If you want to delve deeper into the interplay of SEO, GEO, and machine-readable visibility, you'll find more information in our article on... AI visibility for SMEs the strategic classification. And if you're unsure whether the signals on your existing website are technically sound, a thorough analysis is recommended. Advisory usually cheaper than weeks flying blind.
FAQ: Frequently Asked Questions from Practice
Does robots.txt remove a page from the Google index?
No, robots.txt primarily controls crawling, not the safe removal from the index. If Google already knows the URL, it can still appear, even without visible page content.
Should I use noindex and Disallow together?
Only proceed with extreme caution. If Google is not allowed to crawl the page due to a Disallow directive, it often cannot read the noindex signal. For deindexing, an accessible noindex directive is generally the cleaner approach.
When is the X-Robots tag more useful than a Meta-Robots tag?
This applies whenever you're not controlling a standard HTML page, for example with PDFs, images, or server-side delivered files. The X-Robots tag sits in the HTTP header and gives you significantly more control.
Does every company website already need an llms.txt file?
No. For many SMEs, a clean information architecture, good technical SEO, and clear content are more important than an additional file. If you want to delve deeper into AI-oriented web architecture, our article on... llms.txt and AI-oriented web architecture It would be a good next read.
