Where Does ChatGPT Get Its Data: LLM Pre-Training Corpora, Web Crawling Pipelines, and Real-Time RAG Retrieval Architecture

ChatGPT draws on three kinds of data: a very large static corpus the model was pre-trained on, licensed and curated sources added on top, and live web retrieval at the moment a question is asked. Each layer has a different freshness, a different way of being filtered, and a different route by which a brand's content can enter it. This article describes the three layers, the filtering and deduplication that shape the training corpus, how a retrieval-augmented answer is assembled, and what site owners can check to stay visible.

OpenAI has not published the internals of its production systems. The pipeline described below is a generic reference design for large language model data acquisition and retrieval-augmented generation (RAG), not a description of any one vendor's implementation.

Three Sources of Data Behind a ChatGPT Answer

Frontier autoregressive language models draw on a decoupled, three-tier corpus system.

• Tier 1, static web archives. Large public datasets such as Common Crawl WARC archives, public source-code repositories, academic papers and digitised books. This tier is collected in bulk and then frozen into the model's weights during pre-training.
• Tier 2, licensed and curated sources. High-signal community, technical and publisher content that the model developer obtains through agreements. Reddit, Stack Overflow, Axel Springer and the Financial Times are commonly cited examples of such partners.
• Tier 3, live retrieval at answer time. When a question needs facts newer than the training data, the system searches the live web, reads pages, and injects the best passages into the model's context window. Crawler user agents such as OAI-SearchBot and GPTBot fetch the pages involved.

Raw document streams are filtered before tokenisation. Fast heuristic classifiers judge document quality from signals such as the ratio of punctuation to words, repeated characters and lines, and sentence-length distributions, and they discard spam, machine-generated boilerplate and broken HTTP responses. A normalisation pass then converts heterogeneous markup into a uniform text form such as CommonMark, strips non-semantic page structure, and keeps the canonical URL of each document. How the surviving text is split into model inputs is covered in What Is Tokenisation.

Four-Phase Ingestion and Retrieval Protocol

The four phases below follow a document from the open web to an answer.

1. Phase 1: Distributed corpus harvesting. Crawlers such as OAI-SearchBot, GPTBot and the Common Crawl crawler traverse public hosts within the limits set by the Robots Exclusion Protocol (RFC 9309) and store successful responses as Web ARChive (WARC) records.

2. Phase 2: Quality filtering and deduplication. Heuristic filters and language identification remove boilerplate, error pages and link farms. MinHash locality-sensitive hashing compares shingles of text and removes near-duplicate documents, which shrinks the raw crawl into a much smaller, cleaner training set such as FineWeb (Penedo et al.).

3. Phase 3: Curated and licensed data fusion. Structured, high-density material such as question-and-answer archives, community threads, preprints and journalism is normalised into the same token stream. The model is then trained with next-token prediction, whose loss is:

   $\mathcal{L}_{\text{pre}}(\theta) = -\sum_{t=1}^T \log P_\theta(x_t \mid x_{<t})$

4. Phase 4: Dynamic retrieval and rank fusion. When a question needs grounding beyond the model's knowledge cutoff, the system retrieves from a live web index and, in some designs, from a dense vector index as well. The ranked lists are merged with Reciprocal Rank Fusion (RRF) using a smoothing constant of 60:

   $\text{RRF}(d) = \sum_{m \in M} \frac{w_m}{k + r_m(d)}$

   A late-interaction reranker such as ColBERT can then reorder the fused candidates before the best passages are placed in the context window.

Operating Limits: Crawl Politeness, Latency and Knowledge Cutoffs

Training-data harvesting and live retrieval operate under different constraints.

Harvesting is bound by crawl etiquette. Crawlers follow robots.txt directives under RFC 9309 and keep their concurrency low enough not to overload an origin. A scheduler adjusts its request rate to server response time, connection overhead and crawl-delay hints, so a slow or unreachable site is fetched less.

Live retrieval is bound by latency. A conversational answer is delivered to a user who is waiting, so the retrieval step has a strict time budget. It combines an external index lookup, an optional dense-vector search and a rerank, and the total must stay within that budget:

$T_{\text{total}} = T_{\text{DNS}} + T_{\text{TCP}} + T_{\text{index}} + T_{\text{vector}} + T_{\text{rerank}} \le T_{\text{budget}}$

If retrieval runs over budget, the answer has to be generated from what is already available, which means cached material or the model's own training data.

Knowledge cutoffs are the third limit. A model only knows what was in its corpus up to its cutoff date. Anything that happened afterwards, such as a rebrand, a product launch or a leadership change, exists in an answer only if live retrieval finds it.

Publishers can also restrict what is collected. Sites that block AI crawlers in robots.txt can be excluded from real-time ingestion except through approved syndication feeds.

Fusing Lexical and Dense Rankings with Reciprocal Rank Fusion

Quantifying relevance in a hybrid retrieval system means combining lexical matching with dense semantic embeddings. Term-frequency formulations such as BM25 give a baseline for lexical overlap, while dual-encoder models map documents into a continuous space in which cosine similarity measures conceptual closeness. These scores live on different scales and cannot simply be added.

Reciprocal Rank Fusion solves the scale problem by using only rank positions. Given a set of retrievers $M$ and the rank $r_m(d)$ of document $d$ under retriever $m$, the fused score is the weighted sum of reciprocal ranks, with a smoothing constant $k$ traditionally set to 60. Because only ranks matter, an outlier score from one retriever cannot dominate the merged list. The method is described by Cormack et al. in the paper listed in the references.

Related concepts used in this area include the Common Crawl WARC format, the FineWeb curated corpus, deduplication by MinHash, heuristic text filtering, perplexity-based filtering, reinforcement learning from human feedback (RLHF), direct preference optimisation (DPO), RAG, HNSW and IVF-PQ vector indexes, click-feedback reranking, the knowledge-cutoff boundary and answer engine optimisation (AEO).

Diagnosing Missing and Wrong Brand Mentions

Two structural failure modes explain most cases where ChatGPT misses or misstates a brand.

The first is the training-data boundary. When a company changes its positioning, renames a product or replaces an executive after a model's cutoff, a model that does not retrieve live data will produce a plausible but outdated account from its stored weights. The remedy is to make sure fresh, crawlable pages exist so that live retrieval can find the current facts.

The second is zero-click displacement: a generative summary can use a page's facts without sending the reader to the page. To understand citation loss, audit your server access logs for the OAI-SearchBot and GPTBot user agents, look for HTTP 403 responses, and confirm that structured markup parses cleanly on every public endpoint. Measuring where a brand actually appears in answers is covered in Tracking Brand in AI Responses, and a practical checklist is in How Do I Show Up in ChatGPT.

Semantic entropy over the entities $E_k$ that a context $c$ could refer to gives one way to flag an unstable answer, since high entropy means the model is spread across several candidate entities:

$H_{\text{semantic}}(c) = -\sum_{k=1}^K P(E_k \mid c) \log_2 P(E_k \mid c)$

Standards for Crawler Access and Machine-Readable Entities

Visibility to AI crawlers depends on a few open standards.

RFC 9309 defines the Robots Exclusion Protocol, the robots.txt format in which a site grants or denies access to named crawlers such as OAI-SearchBot. RFC 9110 defines HTTP semantics, including content negotiation, entity tags (ETag) for conditional requests, and caching directives, so servers that implement them correctly make recrawling cheaper for everyone.

Schema.org vocabularies, embedded as JSON-LD, give crawlers explicit facts that do not depend on guessing at page layout. Marking up TechArticle, Dataset and Organization entities supplies a small knowledge graph that entity-disambiguation steps can use during retrieval.

References

• Robots Exclusion Protocol (RFC 9309) — IETF, 2022. https://www.rfc-editor.org/rfc/rfc9309
• HTTP Semantics (RFC 9110) — IETF, 2022. https://www.rfc-editor.org/rfc/rfc9110
• TechArticle — Schema.org. https://schema.org/TechArticle
• Dataset — Schema.org. https://schema.org/Dataset
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al., NeurIPS.
• The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale — Penedo et al., Hugging Face.
• ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT — Khattab and Zaharia, ACM SIGIR.
• Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods — Cormack et al., ACM SIGIR.
• Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs — Malkov and Yashunin, IEEE TPAMI.