Common Crawl

Common Crawl is an American nonprofit organisation that crawls the public web and freely shares its archive. Founded in 2007 by Gil Elbaz, it has published open crawl data since 2008 — more than 10 petabytes to date — and is the dominant raw source from which training datasets for large language models are built.^[1][2]

History

Gil Elbaz, who previously co-founded Applied Semantics (acquired by Google in 2003 and folded into AdSense), founded Common Crawl in 2007 with the goal of making web-scale crawl data available to researchers who lacked the infrastructure to crawl the web themselves. The organisation began archiving sites that year, started publishing regular crawls in 2011, and has run approximately monthly crawls ever since, each typically containing more than two billion web pages.^[1][2]

For most of its history Common Crawl was funded almost entirely by the Elbaz Family Foundation Trust; from 2023 it began receiving donations from the AI industry as demand for pretraining corpora grew. The organisation states that no donor — corporate or otherwise — has any control over what it collects, publishes or removes.^[2][3]

Key Concepts and Technology

Common Crawl's crawler, CCBot, collects publicly accessible web pages and stores them in raw WARC (Web ARChive) format alongside URL and page metadata. Access is free — the archive is hosted on Amazon Web Services — though users pay their own storage and compute costs to process it.^[2] The homepage reports over 300 billion pages spanning 15 years, with 3–5 billion new pages added each month and citations in more than 10,000 research papers.^[4]

Raw crawl data is only the starting point. Because the web contains spam, boilerplate, low-quality text and copyrighted material, researchers build curation pipelines that extract readable text, identify language, deduplicate near-identical documents and filter by quality. This produced the datasets that actually train modern models: Google's C4 (from T5), the Pile-CC component of EleutherAI's The Pile, Meta's RefinedWeb, Hugging Face's FineWeb and FineWeb-Edu, and NVIDIA's Nemotron-CC — a 6.3-trillion-token reworking that combined classifier ensembles and synthetic rephrasing of Common Crawl text.^[5] The same corpus underpins time-continual benchmarks such as TiC-LM, which slices 114 monthly dumps from 2013 to 2024 to study model knowledge staleness.^[6]

Applications and Impact

Common Crawl sits at the base of the AI supply chain: general-purpose pretraining for open models such as Llama and Gemma draws heavily on Common Crawl-derived mixes, while domain-specific mining — code, mathematics, question answering — is typically done by filtering the same archive.^[5] Beyond AI, it supports computational social science, search research, journalism and legal scholarship, giving parties without crawler infrastructure access to the historical web. It also raises ongoing questions about copyright, consent and opt-out — publishers can and do block CCBot — and about bias, since the crawl over-represents the English-language, well-indexed web.^[3]

>See Also

• Large Language Models
• Data Pipeline
• Open-Weight Models
• Synthetic Data
• ILMU