How Search Engines Work: Crawling, Indexing, Ranking, and Neural AI Retrieval Architecture

Modern search engine architecture combines classic distributed crawling and inverted-index ranking with dense vector retrieval, approximate nearest-neighbour graphs and, increasingly, generative answer synthesis. This article walks through the stages a query passes through, the constraints that bound each stage, how ranked lists from different retrievers are merged, and the standards that govern how content is fetched and described.

From Crawl to Ranked Results: The Retrieval Pipeline

Crawling engines schedule HTTP fetch jobs within a crawl budget allocation, and headless browser rendering executes the client-side scripts that build dynamic pages. Postings lists with skip pointers are the foundation of inverted-index term matching: they map each lexical term to a sorted list of document identifiers and positional offsets, which makes candidate generation fast.

A dense retrieval channel runs beside the lexical one. Documents and queries are mapped into a shared embedding space, and approximate nearest-neighbour search over a Hierarchical Navigable Small World (HNSW) graph returns the passages closest to the query. Quantisation techniques such as IVF-PQ compress those vectors so that very large collections fit in memory. Mapping queries and passages into the same metric space is what addresses vocabulary mismatch, the failure of exact-term matching when a query and a relevant passage use different words.

Candidate generation and multi-stage ranking run as a four-phase hybrid fusion protocol: query ingestion, dual candidate retrieval, reciprocal rank aggregation and top-K emission.

Multi-Stage Hybrid Fusion Protocol

1. Phase 1: Ingestion and intent classification. The routing engine parses the incoming query as a lexical, semantic or entity-lookup request and assigns candidate quotas to each channel accordingly. Parsing is kept inside a small, bounded latency budget.
2. Phase 2: Dual-channel candidate traversal. The lexical channel scores distributed inverted postings with BM25, which applies term-frequency saturation and length normalisation (commonly parameterised with $k_1 = 1.2$ and $b = 0.75$). The dense channel queries an HNSW proximity graph using quantised embeddings.
3. Phase 3: Reciprocal rank aggregation. The two ranked lists are merged into one using only rank positions, as described in the section on reciprocal rank fusion below.
4. Phase 4: Top-K reranking and delivery. The fused pool of top candidates is passed to a late-interaction neural reranker such as ColBERT, which produces the final ordering for the results page.

Indexing decouples lexical tokens from dense semantic embeddings, so a single document is represented in both ways. An inverted index is an algorithmic mechanism for mapping lexical terms to sorted integer postings lists that hold document identifiers and positional offsets. Late-interaction models such as ColBERT (Khattab et al., ACM SIGIR 2020) keep one embedding per token and compare query and passage tokens individually, which preserves lexical specificity while still absorbing continuous semantic representations.

Operating Limits: Crawl Budget, Latency and Memory

Every stage of the pipeline runs inside a boundary set by throughput, latency or memory. Understanding the boundary explains most of the practical failure modes technical SEO teams meet.

Crawl budget and server response time

Crawlers allocate a finite fetch budget to each host. When an origin responds slowly, the crawler fetches fewer pages in the same time, so server response latency (TTFB) directly limits how much of a site is crawled. Conditional requests help: when an origin returns 304 Not Modified for unchanged resources, as described by the HTTP caching semantics in RFC 9110, the crawler spends its budget on pages that changed instead of re-downloading pages that did not.

Memory footprint of vector search

Dense vectors are large. A 1536-dimensional embedding stored as 32-bit floats occupies 6,144 bytes, and a collection of many millions of them quickly outgrows memory. Product quantisation (PQ) compresses each vector into a short code, which cuts memory per vector by a large factor at the cost of some quantisation noise. That noise is why systems usually calibrate the result of the first retrieval stage with a more precise second-stage reranker. The neighbour-connectivity parameter of an HNSW graph, usually called M, trades recall and search speed against memory, and the original HNSW work (Malkov and Yashunin, IEEE TPAMI 2020) shows that hierarchical graphs give approximately logarithmic search complexity when connectivity is kept bounded.

Behavioural signals

Ranking also uses click and interaction data. A click-feedback component such as NavBoost learns from aggregated user click streams over a long historical window, using counterfactual learning to rank to correct for position bias. Retrieval relevance alone does not decide the final ordering.

Combining Lexical and Dense Rankings with Reciprocal Rank Fusion

Hybrid retrieval has to merge ranked lists whose raw scores are not comparable: a BM25 score and a cosine similarity live on different scales. Reciprocal rank fusion (RRF) is a ranking algorithm that combines ranked lists from several retrieval systems without any score normalisation. It was introduced by Cormack et al. (ACM SIGIR 2009). For a document $d$ and a set of retrieval models $M$, the fused score is:

$\mathrm{RRF}(d) = \sum_{m \in M} \frac{w_m}{k + r_m(d)}$

Here $r_m(d)$ is the rank of document $d$ in the list produced by model $m$, $w_m$ is an optional weight for that model, and $k$ is a smoothing constant. A common choice is $k = 60$, which damps the influence of any single very high rank and so suppresses outlier variance. Weights let an operator favour one channel: a system that trusts its dense channel more than its lexical one gives it a larger $w_m$. Because RRF uses ranks only, adding a retriever never requires recalibrating the scores of the others.

The table compares the main retrieval approaches along the dimensions that matter when a generative answer engine consumes the results.

Retrieval-augmented generation (Lewis et al., NeurIPS 2020) is the downstream consumer of these ranked lists: the generator is conditioned on retrieved passages, so better passage ranking gives better grounding for the synthesised answer.

Diagnosing Crawl, Index and Retrieval Failures

Three failure families account for most lost visibility in a retrieval pipeline: crawl budget starvation, index-level canonical fragmentation and vector semantic drift. A fourth, zero-click answer synthesis, affects how much of the visibility that does exist turns into attribution.

Crawl starvation and canonical fragmentation

Slow origins starve the crawler of budget, and duplicate parameterised URLs without canonical declarations split ranking signals across near-identical pages. Search engines use near-duplicate detection, for example SimHash fingerprints, to cluster such pages, so a clear canonical URL declaration tells the indexer which version to keep. Fixing server response time and declaring canonical URLs restores crawl capacity and consolidates index entries.

Autonomous agents and robots.txt

The Robots Exclusion Protocol (RFC 9309) is advisory: it tells well-behaved crawlers what they may fetch, but it is not an access-control mechanism, and some autonomous agents ignore it. Sites that need to refuse specific agents enforce the decision at the edge, for example by returning an HTTP 403 Forbidden response after inspecting the user agent.

Semantic drift and answer attribution

Dense embeddings that do not match the vocabulary of a domain produce weak grounding scores, and the usual remedy is to retrain or fine-tune the encoders for that domain. In generative answers, a document can be retrieved but not cited when the synthesis step does not attribute it. Concise direct-answer passages that state the answer plainly are easier for a synthesis step to quote and attribute.

Standards for Crawling and Machine-Readable Markup

Several public standards govern how content is fetched and how it is described to machines. The Robots Exclusion Protocol (RFC 9309) defines the syntax crawlers parse in robots.txt. HTTP Semantics (RFC 9110) defines caching, conditional requests and the status codes crawlers rely on. Schema.org vocabularies, serialised as JSON-LD, supply machine-readable descriptions of an entity and its relationships.

Structured data matters for neural retrieval because it turns unstructured page content into explicit statements about entities. Knowledge graph entity disambiguation is the process of linking an ambiguous text mention to an authoritative identifier, and well-formed markup gives that process unambiguous input. Combining a consistent JSON-LD @graph with correct robots.txt and caching behaviour makes a site easier for search engines and answer engines to crawl, index and attribute.

References

• Robots Exclusion Protocol (RFC 9309) — IETF, 2022. https://www.rfc-editor.org/rfc/rfc9309
• HTTP Semantics (RFC 9110) — IETF, 2022. https://www.rfc-editor.org/rfc/rfc9110
• TechArticle — Schema.org. https://schema.org/TechArticle