Crawling, Indexing, Ranking
Leafline adds a product page for the ZZ plant on Monday. On Tuesday Asha searches for it and finds nothing. Is something broken? Maybe not. The page may simply still be moving through the three stages every search engine runs.
The rule
Crawl to discover, index to remember, rank to answer.
- Crawl. A program called a crawler (also a bot or spider) fetches pages, reads their links and adds new addresses to its list of places to visit. It discovers pages through links from pages it already knows and through sitemaps, files that list a site's URLs. It can only fetch pages it is allowed to (the site's
robots.txtfile sets the rules) and pages that actually load. - Index. The engine processes what it fetched: it reads the text, often runs the page's JavaScript to see the final content, works out the topic, records the links and, where several URLs show the same content, picks one as the main version. What it keeps goes into the index, a vast catalogue of pages. Not every crawled page makes it in; thin or duplicate pages are often left out.
- Rank. When someone searches, the engine pulls matching pages from the index and puts them in order. That ordered list, with ads and other features around it, is the SERP: the search engine results page.
The stages form a chain. A page that was never crawled cannot be indexed, and a page that is not in the index cannot rank.
A quick site:leafline.example search gives a rough idea of what is indexed, but only the engine's own tools, such as Google Search Console, give a reliable answer for a specific URL.
The shortcut
When a page is missing from results, find the first broken link in the chain. Each stage has its own check.
Crawled?
Search Console's URL Inspection shows Google does not know the URL yet. It is linked only from a category page three clicks deep.
The trap: working on rankings for a page that has not been crawled or indexed yet. Always confirm the earlier stages first.
Drills
Answer fast — trust the rule.
In what order do the stages happen?
A page is blocked by the site's robots.txt file. Which stage fails first?
The crawler fetched a page but left it out of the index as a duplicate of another URL. Where did the chain break?
What is the SERP?
Remember
- Crawl to discover, index to remember, rank to answer.
- Missing page? Find the first broken link in the chain.
- The trap: chasing rankings for a page that was never crawled or indexed.