If you want to understand the difference between robots.txt and meta robots , a clear distinction is sufficient: robots.txt controls crawling , meta robots and the X-Robots tag control indexing and, in some cases, the display in the search engine , and llms.txt is, at best, a voluntary aid for AI discovery . These three levels are often mixed up in SME projects. The result is pages that are blocked but still appear, or pages with a noindex directive that Google can never read.
I've seen this for years in website relaunches, e-commerce platforms, and multilingual company websites: the technology is often not the real problem. The problem is that one signal is mistaken for accessibility, another for visibility, and a third for AI orientation. If you clearly separate these, you avoid typical SEO mistakes, unnecessary ambiguity in the index, and a lot of time spent troubleshooting.
Key point: robots.txt tells a search engine what it is allowed to crawl. Meta-Robots tells a search engine what should happen to a specific URL in the index. llms.txt voluntarily describes content for AI systems, but is not a legally binding access control measure.
robots.txt and Meta-Robots: What is the difference?
The comparison between robots.txt and Meta-Robots is fundamental work. Both signals intervene at different points, have different ranges, and perform different tasks.
robots.txt: Purpose, location, effect, boundaries
- Purpose: robots.txt controls the crawling. You give bots instructions on which areas they should or should not access.
- Location of implementation: The file is located in the root directory of a domain. It works with information such as... User agent and Disallow.
- Effect: The file can restrict access to directories, file types, or URL patterns for specific crawlers.
- Limits: robots.txt is no A reliable method to remove a URL from the Google index. However, a blocked URL may still remain known and appear in search results.
Meta-Robots and X-Robots tag: Purpose, location, effect, limits
- Purpose: Meta-Robots It primarily controls the indexing of individual pages. Typical settings include: noindex or nofollow.
- Location of implementation: The meta robots tag is located in the HTML head of a specific page. X-Robots Day In contrast, it is set in the HTTP header and is therefore also practical for PDFs, images or other non-HTML files.
- Effect: This tells a search engine whether a URL should be indexed and how to handle signals from that URL.
- Limits: The search engine must be able to retrieve the resource in order to even see the meta robots tag or X robots tag.
llms.txt: Purpose, location, effect, limits
- Purpose: llms.txt is intended to provide AI systems with a curated overview of important content on a website.
- Location of implementation: Usually located in the root directory of the domain as a separate file.
- Effect: llms.txt can the AI Discovery to support, i.e., finding and classifying relevant content for LLM-based systems.
- Limits: llms.txt is not a recognized web standard, not a replacement for robots.txt, and not a robust mechanism for access control or indexing control.
The most important misconceptions briefly explained
1. A URL blocked via robots.txt is not automatically removed from Google.
This is one of the most common misconceptions. Google explicitly documents that a URL can appear in search results despite a robots.txt block if Google knows the URL through other signals, such as internal or external links: Source: Google Search Central . So, robots.txt prevents the crawling of the content, but not necessarily the knowledge of the URL itself.
2. noindex only works if Google is allowed to read the page.
This is also often implemented incorrectly. According to Google, Googlebot must be able to crawl a page to even see a meta robots tag with noindex or an X robots tag ; if the URL is blocked via robots.txt, this rule is not read and is therefore ignored: Source: Google Search Central . In short: Disallow plus noindex is in many cases a bad combination if you really want to remove a URL from the index.
3. llms.txt is not a replacement for robots.txt
There's currently a lot of attention surrounding llms.txt, but it's important to keep it clearly defined. Jeremy Howard and Answer.AI themselves describe the specification as a proposal and informal specification with community input, not as an officially recognized web standard: Source: GitHub . Therefore, llms.txt is more of a voluntary guideline for AI systems than a technical control instrument.
If you want X, use Y.
- You want to limit crawling in certain areas: use robots.txt with clearly defined User agent and Disallow.
- You don't want a specific page to be in the Google index: use Meta robots with noindex , or X-Robots Day and allow crawling until Google has processed the signal.
- You want to control PDFs or other files that don't have an HTML head: Use that X-Robots Day.
- You want to provide AI systems with guidance on important content: use llms.txt as a voluntary addition, not as access protection.
- You really want to protect sensitive content: Use authentication, rights management, or remove the content entirely. Neither robots.txt nor llms.txt are intended for this purpose.
Typical misconfigurations on SME websites
Especially with owner-managed businesses and small teams, I repeatedly see the same patterns in audits. These usually arise not from negligence, but from time pressure, plugin default settings, or a relaunch without clear technical responsibility. That's precisely why, in our web design and development, we examine not only layout and content, but also the signaling logic for SEO, indexing, and machine-readable accessibility.
- noindex and Disallow simultaneously: The page is supposed to disappear, but at the same time it's blocked for Googlebot. As a result, Google often can't read the noindex signal.
- Block staging or test pages only via robots.txt: This isn't real protection. Password protection is much more reliable for preview or customer areas.
- Global nofollow settings without a strategy: This is sometimes activated due to uncertainty, without it being clear which internal signals are thereby weakened.
- llms.txt as a supposed control over AI access: An llms.txt file can provide guidance, but it is not a binding mechanism to technically block bots.
- Plugin configuration without overall view: A search engine does not just evaluate a single tag, but the interplay of crawling, status codes, internal linking, canonicals and index signals.
Practical logic for SMEs: Fewer files, more clarity
My recommendation is almost always the same: Use each file only for its intended purpose. A good website doesn't need excessive technical control, but rather clear responsibilities. Use robots.txt for crawl rules, meta robots or the X-Robots tag for indexing rules, and llms.txt for optional AI guidance.
If you want to delve deeper into the interplay of SEO, GEO, and machine-readable visibility, you'll find a strategic overview in our article on AI visibility for SMEs . And if you're unsure whether the signals on your existing website are technically sound, expert advice is usually more cost-effective than weeks of flying blind.
FAQ: Frequently Asked Questions from Practice
Does robots.txt remove a page from the Google index?
No, robots.txt primarily controls crawling, not the safe removal from the index. If Google already knows the URL, it can still appear, even without visible page content.
Should I use noindex and Disallow together?
Only proceed with extreme caution. If Google is not allowed to crawl the page due to a Disallow directive, it often cannot read the noindex signal. For deindexing, an accessible noindex directive is generally the cleaner approach.
When is the X-Robots tag more useful than a Meta-Robots tag?
This applies whenever you're not controlling a standard HTML page, for example with PDFs, images, or server-side delivered files. The X-Robots tag sits in the HTTP header and gives you significantly more control.
Does every company website already need an llms.txt file?
No. For many SMEs, a clean information architecture, good technical SEO, and clear content are more important than an additional file. If you want to delve deeper into AI-oriented web architecture, our article on llms.txt and AI-oriented web architecture is also a good place to start.