DL S046 · Building a Pretraining Dataset (SlimPajama-Style)

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 46 / 80 · Module M08 · ~2 hours

The story we're starting with

Here's the uncomfortable truth about LLM pretraining that nobody puts on the marketing slide: the model is 20% of the work. The data is 80%. You can copy nanoGPT line-for-line, spin up an A100 for a weekend, and get garbage — because you fed it garbage. Or you can take a merely OK architecture, feed it a clean, deduped, well-mixed 1B tokens, and get a small model that writes coherent English on a single-GPU budget. The difference isn't the transformer. It's the corpus.

This is the session where we stop worshipping architecture and start doing the janitor work. We're going to build a real pretraining dataset from scratch — Common Crawl → text extraction → language ID → quality filter → dedup → tokenize → shard on disk. By the end you'll have a ~/data/pretrain/*.bin you can point Session 053's training script at.

The numbers to keep in your head as we go: Chinchilla says ~20 tokens per parameter is compute-optimal. So a 100M-param model wants ~2B tokens, and our 1B target is on the light-but-still-teachable side for that scale. A raw Common Crawl WARC dump is ~60 TB per month. After extraction, language filter, quality filter, and dedup, that becomes ~200 GB of usable English text — a 300× shrink. Almost all "data engineering for LLMs" is engineering that shrink pipeline.

Prerequisites

• S044 (BPE) and S045 (SentencePiece) — we'll tokenize with a trained tokenizer at the end.
• S016 (Datasets/DataLoaders) — you should already know why we want fixed-size shards on disk instead of streaming raw JSON.

1 · What is Common Crawl, actually?

Common Crawl is a non-profit that has been crawling the public web since 2008 and publishing the raw HTML on S3 for free. Each monthly crawl (CC-MAIN-YYYY-WW) produces:

• ~3 billion webpages
• ~60 TB of raw WARC files (Web ARChive format — a container of HTTP request/response pairs)
• ~15 TB of WET files (plain-text extraction, produced by CC using a generic extractor)
• ~5 TB of WAT files (metadata: links, headers, no body)

A single WARC file is ~1 GB gzipped and holds ~30–50k pages. Every crawl is split into ~80k WARC files. The index is on S3 at s3://commoncrawl/crawl-data/CC-MAIN-2024-30/....

Why you'll never train on raw WARCs directly:

1. Boilerplate. Nav bars, cookie banners, footers, ads — often 80% of the bytes of a page.
2. HTML noise. <script>, <style>, inline CSS, base64-encoded images.
3. Duplicates. The same news article reprinted on 50 sites. The same Wikipedia stub scraped 40 times.
4. Language sludge. ~55% of the web is English. The rest is 100+ other languages.
5. Low-quality text. Auto-generated pages, SEO word salad, spam farms.
6. Toxic / PII content. You don't want your model memorizing SSNs and slurs.

Every "clean" pretraining corpus you've heard of — C4, The Pile, SlimPajama, RedPajama, RefinedWeb, FineWeb — is the output of a filtering pipeline that starts from Common Crawl (or a subset of it) and cuts 90–99% of the raw text.

The point: we're not building "a webscraper". We're building a series of cheap, parallel filters that each throw away 30–70% of what's left. Doing the filters in the right order is the whole art.

2 · The pipeline in one picture

Here's what we'll build. Read left to right.

Rough survival rates on English CC:

So one WARC file (1 GB gzipped input) yields ~5M usable tokens. To hit our 1B-token target we process ~200 WARC files — about 200 GB of downloads. That's a weekend of aws s3 cp on a $0.20/hr machine. Very doable.

3 · Grabbing WARCs and extracting text

3.1 Downloading one shard

Common Crawl is on a requester-pays-nothing S3 bucket. You need AWS credentials only for signing, not for billing.

# One WARC — pick any file from the paths index
aws s3 cp --no-sign-request \
  s3://commoncrawl/crawl-data/CC-MAIN-2024-30/segments/1720763514315.11/warc/\
CC-MAIN-20240719173234-20240719203234-00000.warc.gz \
  ~/data/warc/shard-00000.warc.gz

For a full crawl you download the warc.paths.gz index for that dump (~1 MB), then iterate:

curl -s https://data.commoncrawl.org/crawl-data/CC-MAIN-2024-30/warc.paths.gz \
  | gunzip | head -200 > shard.list
# Now loop through shard.list and download in parallel
cat shard.list | xargs -P 8 -I{} \
  aws s3 cp --no-sign-request s3://commoncrawl/{} ~/data/warc/

Cost check. S3 egress from us-east-1 is free within region. If you run this on a t3.xlarge in us-east-1 ($0.17/hr on-demand) you pay ~$4 for the compute of a full 200-shard pull.

3.2 Parsing WARC records with warcio

# extract_warc.py — reads one WARC, yields (url, html) pairs
from warcio.archiveiterator import ArchiveIterator
 
def iter_warc(path):
    with open(path, "rb") as f:
        for rec in ArchiveIterator(f):
            if rec.rec_type != "response":
                continue
            # Only process HTML
            ctype = rec.http_headers.get_header("Content-Type", "")
            if "text/html" not in ctype.lower():
                continue
            url = rec.rec_headers.get_header("WARC-Target-URI")
            html = rec.content_stream().read()
            yield url, html

15 lines. Every WARC has request, response, and metadata records — we keep only response HTMLs. About 60% of WARC records are responses; the rest are HTTP requests and metadata.

3.3 Extracting the actual article text

Naive BeautifulSoup.get_text() gives you every scrap of nav-bar and JS-generated garbage. The community-standard tool is trafilatura — a well-benchmarked article-extraction library that beats readability.js and boilerpipe on most eval sets.

import trafilatura
 
def extract_text(html: bytes) -> str | None:
    # favor_precision drops borderline content — good for pretraining
    txt = trafilatura.extract(
        html,
        favor_precision=True,
        include_comments=False,
        include_tables=False,
        deduplicate=True,
    )
    return txt  # None if extraction failed / page too short

Chain it:

for url, html in iter_warc("shard-00000.warc.gz"):
    text = extract_text(html)
    if text and len(text) > 200:
        print(json.dumps({"url": url, "text": text}))

This one loop, single-threaded, runs at ~2500 pages/sec on a modern laptop and produces newline-delimited JSON (ndjson) — the format the rest of our pipeline expects.

4 · Language identification with fastText

Facebook's lid.176 model classifies text into 176 languages, ships as a 130 MB binary, and runs at ~50k predictions/sec on CPU. It is the standard for language ID in every LLM pipeline.

wget https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.bin
pip install fasttext

import fasttext
 
_model = fasttext.load_model("lid.176.bin")
 
def lang_id(text: str) -> tuple[str, float]:
    # fastText wants a single line — no newlines
    sample = text.replace("\n", " ")[:1000]
    labels, probs = _model.predict(sample, k=1)
    lang = labels[0].removeprefix("__label__")
    return lang, float(probs[0])

Filter to English with a confidence floor:

lang, conf = lang_id(doc["text"])
if lang != "en" or conf < 0.65:
    continue

Why 0.65 and not 0.90? Empirically: at 0.90 you throw away code-heavy pages (mixed English/code trips the classifier), forum posts with lots of names, and technical docs. At <0.5 you start letting in low-quality mixed-language slop. 0.65 is the sweet spot that C4 and RefinedWeb converged on.

5 · Quality heuristics — the Gopher rules

This is where "make it clean" gets real. DeepMind's Gopher paper published a checklist that everyone since has copied (with tweaks). Here are the core seven, and why each exists:

Implementation is boring loops. Here's the meaty ones:

STOPWORDS = {"the", "of", "a", "and", "in", "to", "is", "for", "on", "with"}
 
def stopword_count(text: str) -> int:
    words = text.lower().split()
    return sum(1 for w in words if w in STOPWORDS)
 
def symbol_ratio(text: str) -> float:
    words = text.split()
    if not words:
        return 1.0
    n_sym = sum(1 for w in words if w in ("#", "-", "…", "...", "•"))
    return n_sym / len(words)

The repetition penalty is the interesting one:

from collections import Counter
 
def top_ngram_fraction(text: str, n: int = 5) -> float:
    words = text.split()
    if len(words) < n:
        return 0.0
    grams = [" ".join(words[i:i+n]) for i in range(len(words) - n + 1)]
    counts = Counter(grams)
    top_gram, top_count = counts.most_common(1)[0]
    total_chars = sum(len(g) for g in grams)
    return (top_count * len(top_gram)) / total_chars

top_ngram_fraction(text, n=5) > 0.15 and you throw the doc out. This one filter alone catches "make money fast make money fast make money fast..." spam farms that dodge every other check.

The full check:

def passes_gopher(text: str) -> bool:
    words = text.split()
    n = len(words)
    if not (50 <= n <= 100_000):
        return False
    mean_wlen = sum(len(w) for w in words) / n
    if not (3 <= mean_wlen <= 10):
        return False
    if symbol_ratio(text) >= 0.1:
        return False
    if stopword_count(text) < 2:
        return False
    if top_ngram_fraction(text, 5) > 0.15:
        return False
    return True

30 lines. Kills ~55% of what survived language ID. Feels harsh; it is exactly the right amount of harsh.

5.1 The C4 curly-brace rule (weirdest, most effective filter of all time)

C4 famously dropped any line that contained {. The reasoning: HTML/JS/template garbage on the web is saturated with curly braces; real English prose almost never contains them. That one rule strips 5–10% of remaining docs and is one of the highest signal-to-effort filters in the entire pipeline.

def has_curly_lines(text: str, max_fraction: float = 0.05) -> bool:
    lines = text.splitlines()
    with_curly = sum(1 for l in lines if "\{" in l or "\}" in l)
    return with_curly / max(len(lines), 1) > max_fraction

(Escape the braces in your prose or MDX will scream — see the MDX gotcha at the bottom.)

6 · Deduplication — where 40% of your corpus really lives

If you skip dedup you will train on the same Wikipedia article 30 times, the same GitHub license 500 times, and the same Terms of Service boilerplate 5000 times. Your model will memorize instead of generalize. Perplexity will look great on training and cratering on eval. This isn't a small effect.

6.1 Exact dedup — 5 lines that shrink your corpus 30%

seen = set()
def is_dup(text: str) -> bool:
    h = hash(text)          # or use blake3 for cross-process consistency
    if h in seen:
        return True
    seen.add(h)
    return False

That's exact document dedup. RAM is your enemy — hashing 100M docs costs ~800 MB of Python set. For bigger corpora you swap set for a bloom filter or a sorted-file merge.

6.2 Line-level exact dedup

Boilerplate lives at the line level ("© 2024 CompanyName · All rights reserved."). Run a second pass at line granularity:

# Two-pass on disk. Pass 1: hash every line, count.
# Pass 2: for each doc, drop lines whose hash-count > K (e.g., 100).

C4 dropped any line seen in more than one document. That's aggressive; RefinedWeb kept it at "≥10 across corpus".

6.3 Near-dedup with MinHash + LSH (Session preview — full impl in S049)

MinHash gives you a probability-of-Jaccard-similarity fingerprint per doc; LSH lets you find candidate pairs in sub-quadratic time. This is what catches "the same news article on 40 sites" — bodies differ in nav bars but the article text is 95% Jaccard-identical.

We'll implement MinHash + LSH from scratch in S049. For today, just know: after MinHash dedup at threshold 0.8, expect another 30–40% shrink.

7 · URL blocklists and PII

The lightest possible filter but you should have it.

• URL blocklist: UT1 blocklist (~450k adult/spam domains) covers 95% of what you don't want. Load once into a set of hostnames, reject on match.
• PII scrub: simple regex for email addresses, phone numbers, credit cards. For a real production run use Microsoft Presidio (fancier NER) but the regex catches the 80% for free.

import re
EMAIL_RE = re.compile(r"[\w.+-]+@[\w-]+\.[\w.-]+")
PHONE_RE = re.compile(r"\+?\d[\d\s\-()]{8,}\d")
 
def scrub_pii(text: str) -> str:
    text = EMAIL_RE.sub("[EMAIL]", text)
    text = PHONE_RE.sub("[PHONE]", text)
    return text

This is deliberately blunt. Real PII detection is a whole field. For a personal pretraining run, this is enough to keep the worst leakage out of your weights.

8 · Tokenize and shard

You trained a tokenizer in S045. Load it and tokenize every surviving doc:

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("./my-bpe-tokenizer")  # your S045 output
EOD = tok.eos_token_id                                     # end-of-doc separator
 
def tokenize_doc(text: str) -> list[int]:
    return tok.encode(text) + [EOD]

Write to fixed-size .bin shards — this is what nanoGPT-style trainers expect:

import numpy as np
SHARD_TOKENS = 100_000_000    # 100M tokens per shard
 
buf, shard_idx = [], 0
for text in stream_docs():
    buf.extend(tokenize_doc(text))
    if len(buf) >= SHARD_TOKENS:
        arr = np.array(buf[:SHARD_TOKENS], dtype=np.uint16)
        arr.tofile(f"~/data/pretrain/shard-{shard_idx:04d}.bin")
        buf = buf[SHARD_TOKENS:]
        shard_idx += 1

np.uint16 gives you up to vocab_size 65535, which is plenty for a 32k or 50k BPE vocab and cuts disk usage 2× vs int32. For a 100k+ vocab (Llama 3, Qwen), bump to uint32.

A 1B-token corpus at uint16 is 2 GB on disk. Not big.

9 · Putting it all together — the driver script

# build_dataset.py — the full pipeline as one streaming iterator
def process_shard(warc_path: str) -> list[dict]:
    out = []
    for url, html in iter_warc(warc_path):
        text = extract_text(html)
        if not text or len(text) < 200:
            continue
        lang, conf = lang_id(text)
        if lang != "en" or conf < 0.65:
            continue
        if is_url_blocked(url):
            continue
        if not passes_gopher(text):
            continue
        if has_curly_lines(text):
            continue
        text = scrub_pii(text)
        out.append({"url": url, "text": text})
    return out

Run in parallel with multiprocessing.Pool — one process per WARC, 8–16 workers on a 16-vCPU box. Wall clock for 200 WARCs on a c7i.4xlarge ($0.72/hr): ~4 hours. Total cost of the corpus: **~$3 in EC2 + $0 in bandwidth =$3 for 1B tokens**.

Feed the surviving stream into the dedup + tokenize + shard stage and you're done.

10 · Diagram — where every byte goes

Every filter is a settling tank. By the time text becomes tokens, ~98.6% of the original download is on the cutting-room floor. That is normal, correct, and by design.

11 · War stories

12 · What we did NOT do (and where to go for it)

• Model-based quality filtering. Fancier corpora (RefinedWeb v2, DCLM, FineWeb-Edu) train a small classifier (fastText on labeled "high quality vs low quality") and use it as a scoring gate. Adds ~5 points to downstream eval scores. We cover this in S049.
• PDF / arXiv / Wikipedia mixing. We built a pure-web corpus. Real pretraining datasets (SlimPajama, RedPajama) mix web + books + arxiv + wiki + code in specific ratios. That's next session's job (S048).
• Toxicity classification. Simple keyword filters miss coded speech; classifiers exist (Perspective API, Detoxify). Beyond scope.
• Legal. Common Crawl is a research corpus. Commercially-licensed models often use much stricter licensing filters. Beyond scope.

stages = ['raw_warc', 'text_extracted', 'lang_filtered',
          'quality_filtered', 'deduped']
counts = {s: 0 for s in stages}

# Instrument the driver: increment at each stage, do not just
# log the final total.
prev = None
for s in stages:
    n = counts[s]
    if prev:
        print(f'{s:20s} {n:>10,}  kept {n/prev:.1%}')
    prev = n

# Then read 20 documents that each stage REJECTED.
# If the quality filter is discarding things you would have
# wanted, your thresholds are wrong. If dedup is removing very
# little, your dedup is broken -- web text is far more
# redundant than it appears.

13 · Retention scaffold

Recall — answer from memory (no scrolling)

The 2024–2025 pretraining-data revolution

S046 as-written teaches you the SlimPajama-style pipeline circa 2023. That pipeline is now the floor, not the ceiling. Three datasets released in 2024–2025 changed what "good pretraining data" means, and you should know their names and numbers before you commit to a mix:

• DataComp-LM (DCLM), Li et al., 2024 (arxiv 2406.11794) — the first benchmark for pretraining data curation. Fixed model architecture (1.4B), fixed compute (28B tokens), only knob = which subset of a 240T-token pool you keep. DCLM-Baseline (3.8T tokens) is a filtered subset of Common Crawl that beat every open pretraining set at matched compute. The filter recipe: fastText classifier trained on OH-2.5+ELI5 as "positives" vs random CC as "negatives". A logistic-regression classifier picked ~10% of raw CC and produced better models than the full corpus. Read this if you read nothing else. https://www.datacomp.ai/dclm/
• FineWeb / FineWeb-Edu (HuggingFace, 2024) — Penedo et al.'s 15T-token clean CC dump + a 1.3T-token "educational content only" subset scored by a Llama-3-70B judge on a 0–5 scale. FineWeb-Edu doubles benchmark scores per training token compared to raw web. Recipe writeup: https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1
• Nemotron-CC (NVIDIA, 2024) — 6.3T-token corpus using LLM-based rewriting to synthesize higher-quality versions of low-quality docs (turn a listicle into a coherent paragraph). Meaningfully improves reasoning benchmarks vs FineWeb-Edu at matched tokens.
• RedPajama-V2 (Together, 2023–2024) — 30T-token CC dump with precomputed quality signals (40+ metrics per doc) so you can filter with pandas in an afternoon instead of running your own classifiers. https://github.com/togethercomputer/RedPajama-Data

The unifying lesson: the frontier of data curation in 2024–2025 shifted from heuristics to learned filters and synthetic rewriting. A fastText classifier trained on a few hundred thousand "good" vs "random" docs replaces dozens of Gopher-style rules and beats them on downstream evals. And you don't need to reinvent this — all four datasets above are on HuggingFace, permissively licensed, ready to datasets.load_dataset(...).

Stretch

Download one WARC file (~1 GB gzipped) from the latest CC dump, run the full pipeline end-to-end (extract → langid → Gopher → exact dedup → tokenize), and report: number of docs in, number of docs out, number of tokens, wall clock, and peak RAM. Sanity check: you should see ~5M tokens survive. If you see 100k or 20M, something is misconfigured.

In your own words

Explain, in one sentence, why data-cleaning matters more than architecture for pretraining a small model:

Spaced-review pointer

• S016 (Datasets/DataLoaders) — the .bin shard format is exactly what you'll mmap in the training loop.
• S044 (BPE) and S045 (SentencePiece) — you're using the tokenizer you trained there.

Next-session teaser

Right — you have 1B tokens. But how many should you have? Session 047 (Chinchilla) tells us the exact ratio of parameters to tokens that spends compute optimally. Spoiler: for a fixed budget, you want ~20 tokens per parameter — and most published models before 2022 were massively under-trained.

What to bring back tomorrow — sticky note

• Diagram: the 6-stage funnel (WARC → extract → langid → quality → dedup → tokens).
• Equation: 1 GB WARC ≈ 5M tokens. Scale from there.
• Snippet: passes_gopher(text) — 30 lines that decide if a doc lives or dies.

Previous: ← DL S045 · Next: DL S047 →