Crawling vs indexing

The two stages

Google's documentation describes three stages for any web page: crawling, indexing and serving search results. Crawling is the download: Googlebot requests the URL and receives the HTML, images and other resources. Indexing is the analysis: Google processes the text and tags, works out which version of duplicate pages is canonical, and decides whether to store the page in its index. Serving is the ranking of stored pages for a query.

Google states plainly that it "doesn't guarantee that it will crawl, index, or serve your page, even if your page follows the Google Search Essentials." Each stage is a separate filter.

Discovery comes first

Before Google can crawl a page it has to know the URL exists. Google calls this URL discovery and notes that most new pages are found "when Google extracts a link from a known page to a new page." A guest post on a deep page of a small blog may have no links from anywhere Google visits often, so it sits undiscovered. Search Console uses the status "Discovered - currently not indexed" for URLs Google knows about but has not yet fetched.

The gap after the crawl

The status that matters most for link builders is "Crawled - currently not indexed." Google's page indexing report describes it as a page that "was crawled by Google but not indexed. It may or may not be indexed in the future; no need to resubmit this URL for crawling." In other words, Google has seen the page and parked it. Repeating the crawl will not change the outcome unless something about the page or its context changes.

Common reasons a crawled page is not stored include:

• The content is thin, auto-generated or near-identical to other pages.
• The page declares a canonical URL elsewhere, or Google chooses another canonical.
• A noindex meta tag or X-Robots-Tag header tells Google not to index it.
• The site as a whole has weak signals, so Google indexes only a fraction of its pages.

The full list, with fixes, is in why backlinks don't get indexed.

Why the distinction decides backlink value

A backlink is a link on someone else's page. If that page is never crawled, Google never reads the link. If the page is crawled but not indexed, Google has fetched it, but the page is not part of the index used for search, and practitioners generally treat such links as weak or uncounted. The debate is covered in does Google count unindexed backlinks. The safe working assumption in the trade is that a link pays off reliably only once its page is indexed.

This is also why a backlink indexer is sold the way it is. The service acts on discovery and crawling, the stages an outsider can influence. Reputable vendors guarantee the crawl and measure indexation as an outcome. IndexChex, which publishes this wiki, takes that position: it guarantees that Googlebot crawls submitted URLs, does not guarantee indexing, and does not refund credits for URLs Google declines to store.

Measuring each stage

For third-party pages, the only independent evidence of indexing is a check against live search results. Techniques are explained in verifying backlink indexation, and the limits of Google's own inspection tool for pages you don't own are in Search Console URL Inspection limits.

Timing

Crawling and indexing also run on different clocks. A crawl can be triggered within minutes. Google's recrawl guidance says crawling "can take anywhere from a few days to a few weeks" when left to its normal schedule, and that requesting a crawl "does not guarantee" inclusion. Indexing follows the crawl after a processing delay that varies by site. Ranges for both are collected in how long backlink indexing takes; the mechanics of triggering the crawl are in how backlink indexers work.

Summary

Crawling is access; indexing is acceptance. Tools and site owners can improve access. Acceptance depends on the page itself and on Google's view of the site, which is why every accurate description of link indexing separates the two.