The website indexing chain describes the technical and editorial requirements a URL goes through from being stored in the CMS to potentially being included in the Google index. If a page doesn't appear in Google, you check this chain in a fixed order: published, internally linked, findable, crawlable, correctly delivered, renderable, and indexable.
In my work with SMEs, I often see that the SEO plugin is initially suspected as the cause. However, the problem often lies elsewhere: the page is not public, not internally linked, marked with noindex , canonicalized to a different URL, or accessible to Googlebot differently than to a regular website visitor.
An accessible URL is not automatically crawled. A crawled URL is not automatically indexed. An indexed URL does not automatically appear in every search query.
The website indexing chain from CMS to Google
Indexing does not depend on a single switch. Several successive signals must match up. An error at the beginning affects all subsequent checks.
Publication and Discovery
- Published in the CMS: The page is public and not just available as a draft or preview.
- Internal link: At least one crawlable page points to the new URL.
- Discoverable: Google can find the URL via links or an XML sitemap.
- Crawlbar: The robots.txt file and other access controls do not prevent the retrieval.
Technical processing and indexing
- Indexable: Meta robots or HTTP headers do not contain an unintentional noindex directive.
- Clearly: The canonical reference points to the desired main version.
- Delivered correctly: The server returns the expected HTTP status 200 for the regular content page.
- Renderable: Relevant content remains visible even after JavaScript processing.
- Processed: Google crawled the URL, evaluated it, and indexed it where appropriate.
Discovered, crawlable, crawled, rendered, and indexed
These terms describe different states and should not be equated:
- Discovered: Google knows the URL from a link, a sitemap, or a previous processing.
- Crawlbar: Googlebot is generally allowed and able to retrieve the URL.
- Crawled: Googlebot made a request to the URL and received a response.
- Rendered: Google processed the page and may have executed JavaScript in the process.
- Indexed: Google has included the URL, or its selected canonical version, in its search index.
Crawling refers to the technical retrieval of data, not its inclusion in the index. This distinction is crucial if you want to test the indexing and determine the actual bottleneck.
Diagnosis by fault level: check, understand, fix
1. Is the page actually published in the CMS?
Check: Open the URL in a private browser window where you are not logged into the CMS. Also check the publication status, the date, and any visibility settings.
Finding: The URL only works in the preview, requires registration, or displays an error page.
Reason: The page is still a draft, private, password-protected, time-controlled, or published under a different URL.
Action: Publish the page normally and then test it again without logging in. A preview URL should not be included in internal linking or the XML sitemap.
2. Is there a discoverable internal link?
Check: Search the website for a normal text, navigation, or teaser link to the new URL. The link should be present in the delivered HTML and not only appear after user interaction.
Finding: The page can be accessed directly, but is not reachable from any other relevant page.
Cause: This is an orphaned page . Google may be able to discover the URL via the sitemap, but receives only weak signals about its context and importance through internal linking.
Action: Link to the page from a relevant overview, service, or topic page. Clear internal linking helps Google and directs visitors to the content.
3. Is the URL included in the XML sitemap?
Check: Open the XML sitemap and search for the exact URL. Pay attention to discrepancies in the domain, HTTPS, language path, trailing slash, and capitalization.
Diagnosis: The page is missing, even though it should be indexed. Or the sitemap contains an old, redirected, or non-indexable version.
Cause: The CMS or SEO plugin has not yet updated the sitemap, the page has been excluded, or multiple systems are generating conflicting sitemaps.
Action: First, correct the publishing, indexability, and canonical tags. Then, update the sitemap and submit the correct XML sitemap to Google Search Console.
Google explains that a sitemap supports URL discovery but does not guarantee crawling or indexing of all contained content. The XML sitemap is a signal, not confirmation of inclusion. This is evidenced in Source 1.
4. Are robots.txt or Meta-Robots blocking processing?
Check: Verify the differences between robots.txt and Meta-Robots . robots.txt primarily controls access by crawlers. A Meta-Robots directive in the HTML or a corresponding HTTP header can prevent indexing using the noindex directive.
Finding: Google is not allowed to crawl the URL or detects a noindex directive on the requested page.
Cause: Such rules often originate from a development environment, a global CMS setting, a page template, or an incorrectly set option in the SEO plugin.
Action: Remove only the rule that contradicts the intended visibility. Do not change robots.txt and Meta-Robots simultaneously on suspicion, as they perform different tasks.
5. Does the canonical tag point to the correct URL?
Check: Search the source code for the canonical tag. Compare the URL specified there with the URL you want to index and with the canonical URL selected by Google in Google Search Console.
Finding: The page is accessible and indexable, but the canonical tag points to a different page.
Cause: A copied template, an incorrect language assignment, URL parameters, a domain change, or a faulty plugin configuration can change the canonical tag.
Action: For independent content, use a consistent, self-referencing canonical URL. In the case of true duplicates, the canonical should explicitly point to the preferred main version. The canonical is a signal; Google chooses the final canonical URL.
6. Does the server return the correct HTTP status?
Check: Retrieve the HTTP status using a header inspector, the browser's developer tools, or via URL inspector. For a regular content page, you would typically expect an HTTP status of 200.
Diagnosis: The URL redirects, returns a server error, is not found, or displays only a content error page despite HTTP status 200.
Cause: Redirect rules, security modules, hosting problems, faulty language logic, or client-side errors can deliver different responses to browsers and Googlebot.
Action: Correct the status where it originates: in the CMS, on the web server, in the upstream cache, or during hosting. A non-existent page should not return an HTTP status 200 like a regular content page.
7. Can Google render the crucial content?
Check: Compare the original source code with the rendered output in the live URL test. Pay particular attention to headings, main text, links, canonical tags, and meta robots.
Finding: Visitors see the content in the browser, but the rendered result from Google is missing text, navigation, or links.
Cause: Content is only loaded via JavaScript, resources are blocked, scripts abort, or important information only appears after a click, after scrolling, or after logging in.
Action: Deliver key content and links in a way that allows for reliable processing. For complex applications, server-side rendering or pre-rendering may be beneficial to ensure that the main content is not entirely dependent on subsequent script execution.
Google describes the processing of JavaScript pages in the phases of crawling, rendering, and indexing. Rendering can be staggered via a queue. Evidence can be found in source 3.
8. Was the page crawled but not indexed?
Check: Open the URL audit in Google Search Console. In addition to the summary status message, also check the crawl time, the canonical tag, the indexability, and the detected sitemap.
Finding: Google retrieved the URL but did not index it. Alternatively, Google selected a different canonical URL.
Cause: Possible causes include pending processing, duplicate or very similar content, low originality, conflicting signals, or a different canonical selection. Therefore, the status "Crawled – not currently indexed" is not a definitive error diagnosis.
Action: First, check for technical inconsistencies. If the technology is sound, evaluate the content: Does the page answer a specific question? Does it offer clear added value? Is its significance evident through navigation and internal links? Only request re-indexing after a verifiable correction has been made.
Check website indexing chain in 45 minutes
To perform the check, you'll need access to the CMS, the Google Search Console, and ideally, the hosting or server logs. Work through the steps in the order given. This will allow you to assign the error to a specific stage, rather than addressing subsequent symptoms.
Minutes 0 to 5: CMS and public URL
- Open the page without logging into the CMS.
- Check the publication status and final URL.
- Check that the title, main content, and navigation are fully visible.
Result: You know whether a real, publicly accessible website exists.
Minutes 5 to 10: Source code and meta-robots
- Open the page source code.
- Search for "noindex" and the canonical tag.
- Compare the canonical URL with the verified URL.
Result: You can see whether the page identifies itself as indexable and which URL it designates as canonical.
Minutes 10 to 15: Internal linking
- Open the most important overarching topic or service page.
- Check if there is a normal link to the target page.
- Evaluate whether the link text describes the content clearly.
Result: You know whether the URL is findable within the website and is categorized according to its subject matter.
Minutes 15 to 20: XML sitemap and robots.txt
- Find the exact URL in the XML sitemap.
- Open the robots.txt file in the root directory of the domain.
- Check if the relevant path is blocked for Googlebot.
Result: You can identify whether discovery or crawling is hindered by central rules.
Minutes 20 to 25: HTTP status and redirects
- Check the HTTP status of the target URL.
- Check if and where the URL redirects.
- Compare redirect target, sitemap entry, and canonical tag.
Result: You know whether the technical signals point to the same URL.
Minutes 25 to 35: URL check and live URL test
- Check the full URL in Google Search Console.
- Note the saved index status and the last crawl time.
- Then perform the live URL test.
- Compare accessibility, indexability, resources, and rendered output.
The standard URL check shows the last saved index status. The live URL test, on the other hand, retrieves the currently accessible version and checks its potential indexability. The results may differ if you have changed the page since Google last accessed it. This is evidenced in Source 2.
Result: You can distinguish between an older index level and a currently existing problem.
Minutes 35 to 42: Evaluate server logs
- Search for queries to the specific URL.
- Filter by the user-agent entry Googlebot.
- Note the time, requested URL, and delivered HTTP status.
- For important diagnostics, have the origin of the crawler technically verified, as the name in the user agent alone is not sufficient proof.
My article on server logs for SMEs provides a practical introduction . Server logs show whether the web server received a request for a specific URL and what HTTP status it delivered.
Result: You can see whether an access marked as Googlebot requested the URL and how the server responded.
Minutes 42 to 45: Assign the finding to an error level
- Not published: Fix it in the CMS.
- Orphaned: Add internal linking.
- Not detected: Correct links and sitemap.
- Blocked: Check robots.txt, Meta-Robots, or access protection.
- Incorrectly canonized: Clean up canonical and duplicate entries.
- Delivered incorrectly: Correct HTTP status, redirects, or hosting.
- Not renderable: Technically examine JavaScript and resources.
- Crawled, not indexed: Evaluate independence, quality, and website architecture.
Result: You have a concrete diagnosis and a prioritized next step.
Why an SEO plugin cannot force indexing
An SEO plugin can help you manage meta robots, canonicals, XML sitemaps, and other metadata. However, it doesn't decide when Google crawls a URL, which canonical tag Google selects, or whether a page is included in the index.
Selecting and correctly using a WordPress SEO plugin can reduce configuration errors. However, the plugin remains a tool within the website indexing chain. A properly configured plugin will not fix an orphaned page, a server error, or a content-independent post.
An SEO plugin can output technical signals. It cannot force crawling, indexing, or rankings.
Practical example: Canonical and internal linking as the cause
At a small, owner-managed company I worked with, a new services page didn't appear in Google's index for an extended period. Internally, it was suspected that the SEO plugin was generating a faulty sitemap. However, the page was correctly listed there, publicly accessible, and returning an HTTP status code of 200.
Examining the website's indexing chain revealed two other causes: The new page was not linked from any regular content page and, due to a copied template, contained a canonical URL pointing to an older, more general performance page. The signals were contradictory: The sitemap listed the new URL, the website architecture treated it as an orphaned page, and the canonical URL pointed to a different main version.
We didn't replace the plugin. We corrected the canonical tag, linked the page from the relevant service overview, and clarified the content boundaries. We then rechecked the URL and monitored the subsequent processing. The consistency of the signals was crucial, not the replacement of a tool.
Requesting indexing: useful after a correction
If you've fixed a verifiable bug or published an important page, you can request indexing in Google Search Console. This request requests re-evaluation but doesn't guarantee inclusion.
Google points out that a crawl or indexing request does not guarantee immediate or later inclusion. Repeated requests for the same URL do not speed up crawling. Evidence can be found in source 4.
Therefore, request indexing only after a verifiable change has occurred. Then, check internal linking, content, and technical signals instead of submitting the same request repeatedly.
Frequently Asked Questions about Crawling and Indexing
How long does it take to index a new page?
There is no reliable deadline for indexing. A technically sound and well-integrated website creates good conditions, but inclusion cannot be scheduled or guaranteed.
Why is a URL not indexed despite having an XML sitemap?
The XML sitemap aids discovery. Additionally, check internal links, robots.txt, meta robots, canonical tags, HTTP status, rendering, and the score in Google Search Console.
What does "Crawled - not currently indexed" mean?
Google has accessed the URL, but it's not currently indexed. Check for technical inconsistencies, similar pages, the canonical tag, the content's independence, and the page's relevance within your website.
When is a live URL test useful?
The live URL test is particularly useful after you've resolved a noindex directive, a robots.txt block, a server error, or a rendering problem. The test shows the currently accessible version but does not replace an indexing decision.
How can I identify Googlebot in the server logs?
Filter the server logs for the checked URL, the timestamp, and the Googlebot user-agent entry. For a reliable diagnosis, a technical administrator should also check the origin of the access and the delivered HTTP status.
Can an SEO plugin force indexing?
No. An SEO plugin manages signals such as XML sitemap, canonical and meta robots, but cannot force crawling, indexing or rankings.
Is an accessible page automatically crawlable?
No. A page can be visible to you in your browser, but Googlebot can still block it due to robots.txt, access restrictions, security rules, or inconsistent server responses. Therefore, compare browser access, live URL tests, and, if necessary, server logs.
Should every published CMS page be indexed?
No. Thank-you pages, internal search results, filter variations, weak archives, or actual duplicates do not need to be indexed. What matters is whether the URL offers independent value to searchers.
When you can take action yourself and when technical help is needed.
Self-correctable content or configuration error
You can usually take action yourself if a page is unpublished, not internally linked, unintentionally set to noindex, or is barely distinguishable from an existing page in terms of content. Document every change and don't edit multiple levels simultaneously.
Task for web development or hosting
Technical support is advisable when HTTP status, redirect chains, server-side canonicals, JavaScript rendering, access control, caching, or server logs are affected. In our web development and website strategy , we examine content, architecture, and technical delivery as a cohesive system.
Case for a quality or architectural review
If many URLs have the status "Crawled – not currently indexed," Google regularly selects different canonicals, or important pages are only findable via the sitemap, a single fix is often insufficient. In such cases, systematically examine templates, information architecture, internal linking, content overlaps, and the independent value of the pages.
First, check the broken stage and only then change the affected tool or system. A consistent website indexing chain makes technical errors traceable and prevents changes made based on suspicion.
Sources
- Google Search Central: Learn about sitemaps — developers.google.com (2025)
- Google Search Console Help: Inspect and troubleshoot a single page — support.google.com (n.d.)
- Google Search Central: Understand the JavaScript SEO basics — developers.google.com (2026)
- Google Search Central: Ask Google to recrawl your URLs — developers.google.com (2025)
