Pretraining data: auditable extraction, grouping and token shards

The corpus is an artifact, not a token total

A useful pretraining pipeline turns permitted source records into versioned training examples whose provenance, splits, filters, token IDs and boundaries can be reconstructed. Copying an architecture is not enough, but “data is80% of the work” is an anecdotal slogan, not a measured allocation law. We will build auditable local stages and a complete opt-in WARC adapter, not claim that a1B-token public-web corpus was collected here.

Bring tokenization from DL044–045 and dataset iteration from DL016. Outcomes: distinguish WARC/WAT/WET; extract local HTML response records; report language/filter decisions; separate rights, privacy and deduplication; group before splitting; write bounded token shards without truncating tails or overflowing IDs; estimate cost from a measured pilot rather than a fixed cloud price.

1 · Common Crawl and the extraction boundary

WARC is an archival record container used for web crawl content and metadata, not merely a list of plain-text documents. Records can include responses, requests, metadata and revisits. A response may contain HTML, an error page or another MIME type. WAT contains derived metadata; WET contains extracted text. WET is convenient but gives up control over the original HTML extraction choices. Crawl releases vary in page counts, compressed sizes and composition; no universal60TB/month,55%-English or5M-tokens-per-WARC conversion is defensible.

Extraction must remove unwanted navigation/script/style/boilerplate while preserving useful article text, tables and code according to the target domain. A precision-oriented extractor can discard useful technical content. Compare extraction settings on the same sample and inspect keeps and rejects, rather than promise that changing one flag retains20–40% more documents.

Common Crawl's public access does not require adding credentials when anonymous HTTP or --no-sign-request access is supported. Compute, storage and transfer policy still depend on your environment; “publicly downloadable” does not imply zero total cost or a blanket right to train commercially. No download command is executed by this lesson's review.

2 · A manifest and measured funnel

For each source retain a stable document ID, source URI, crawl/version/date, usage basis or unresolved-rights flag, raw content hash, extraction/version configuration, parent lineage and transformation decisions. Never log private raw records indiscriminately; retention/access rules apply to reject samples too. A public dataset card's code/distribution license need not settle underlying document rights.

The diagram describes dependencies, not one universally cheapest execution order. Fixed heuristics can be applied before grouping, but learned filters/tokenizers must not be fitted on evaluation records. Cross-split duplicate discovery may require examining content solely to group/exclude overlaps under a predeclared leakage policy; it is not permission to tune thresholds on test scores. A pre-existing external benchmark is screened separately.

Record counts of input records, HTML responses, extracted documents, characters/UTF-8 bytes, accepted/rejected records by reason, unique groups and final tokens. Compressed WARC bytes, uncompressed text bytes and token-array bytes have different units. Do not label 20MB text →5M uint16 tokens as1.4% of a1GB input: token payload is10MB under those assumptions, and the assumptions require measurement.

3 · Exact-duplicate audit: preserved small program

The following verified authored-text example remains byte-identical. Its canonicalization folds case and whitespace only for duplicate keys; original text remains intact. It is inappropriate for case-sensitive code unless that policy is intentional. Cryptographic equality detects normalized duplicates, not semantic equivalence, PII or rights compliance.

import hashlib
 
records = [
    {'id': 'a', 'text': 'Red birds fly.', 'basis': 'authored toy'},
    {'id': 'b', 'text': ' red   birds fly. ', 'basis': 'authored toy'},
    {'id': 'c', 'text': 'Blue fish swim.', 'basis': 'authored toy'},
    {'id': 'd', 'text': 'Green trees grow.', 'basis': 'authored toy'},
]
 
def key(text):
    normalized = ' '.join(text.lower().split())
    return hashlib.sha256(normalized.encode()).hexdigest()
 
groups = {}
for record in records:
    groups.setdefault(key(record['text']), []).append(record)
assert len(groups) == 3
assert any(len(group) == 2 for group in groups.values())
# Tiny illustrative group split, not a statistically adequate evaluation set.
ordered = sorted(groups)
splits = {'train': ordered[:1], 'validation': ordered[1:2], 'test': ordered[2:]}
assert not (set(splits['train']) & set(splits['test']))
print('documents', len(records), 'duplicate groups', len(groups))

Related documents can differ in hashes: chapters of a book, messages from one conversation, syndicated versions and generated templates may be dependent. Group these before splitting. Near-duplicate grouping is developed in DL049; a deterministic exact-hash pass remains useful rather than being dismissed as irrelevant. Preserve duplicate→retained lineage instead of silently deleting records.

Line-level repetition needs a separate two-pass policy. First count in how many distinct documents each normalized line appears, not how many times one document repeats it. Then record candidate line removals above a chosen document-frequency threshold, preserve original line indexes and inspect whether removal creates broken sentences or code. The following marks repeated boilerplate without modifying the originals; it is not a reproduction of C4's selection policy.

def repeated_line_candidates(items, minimum_documents=2):
    if minimum_documents < 2: raise ValueError('need at least two documents')
    document_frequency = {}
    for doc_id, text in items:
        for line in set(' '.join(line.split()) for line in text.splitlines() if line.strip()):
            document_frequency.setdefault(line,set()).add(doc_id)
    return {doc_id: [i for i,line in enumerate(text.splitlines())
                    if len(document_frequency.get(' '.join(line.split()),())) >= minimum_documents]
            for doc_id,text in items}
 
line_fixture=[('a','article one\nCopyright example\nCopyright example'),
              ('b','article two\nCopyright example'),('c','different material')]
assert repeated_line_candidates(line_fixture)=={'a':[1,2],'b':[1],'c':[]}
assert repeated_line_candidates([line_fixture[0]])=={'a':[]}

4 · Heuristics with explicit definitions

A quality heuristic is a feature and a threshold, not proof of truth or safety. Typical features include word count, average word length, symbol density, bullet/ellipsis lines, stop-word presence and repeated spans. They can reject valuable short answers, dialects, multilingual text, formulas and source code. Calibrate by domain and inspect borderline examples.

The original lesson called its partial implementation “the Gopher rules” while changing definitions. In Gopher's appendix, shorter n-grams and longer duplicated n-grams use different character-fraction measurements. A top5-gram count divided by the sum of lengths of all overlapping5-grams is not the fraction of document characters occupied by repeats. The corrected function below unions repeated-word positions, so overlap is not counted multiple times. It is an explicit teaching policy, not an exact Gopher reproduction.

C4's cited TensorFlow Datasets source returns from document filtering when it encounters an opening curly brace in a qualifying line; that is not simply “drop that line.” C4 also has a separate duplicate-line procedure. A brace gate can remove legitimate JSON/code/math and is not near-zero-false-positive. The old Python string '\\{' checks for a literal backslash plus brace, not an ordinary brace.

import re
from collections import Counter
 
STOPWORDS = {'the','of','a','and','in','to','is','for','on','with'}
 
def quality_signals(text, ngram=5):
    words = text.split(); lines = [line for line in text.splitlines() if line.strip()]
    n = len(words)
    grams = [tuple(words[i:i+ngram]) for i in range(max(0,n-ngram+1))]
    counts = Counter(grams); covered = set()
    for i,g in enumerate(grams):
        if counts[g] > 1:
            covered.update(range(i,i+ngram))
    char_total = sum(map(len,words))
    return dict(words=n,mean_word_chars=char_total/n if n else 0.,
        symbol_word_fraction=sum(w in {'#','-','…','...','•'} for w in words)/max(n,1),
        bullet_line_fraction=sum(l.lstrip().startswith(('-','*','•')) for l in lines)/max(len(lines),1),
        ellipsis_line_fraction=sum(l.rstrip().endswith(('...','…')) for l in lines)/max(len(lines),1),
        stopword_occurrences=sum(w.lower().strip('.,;:!?') in STOPWORDS for w in words),
        repeated_word_character_fraction=sum(len(words[i]) for i in covered)/max(char_total,1),
        brace_line_fraction=sum('{' in l or '}' in l for l in lines)/max(len(lines),1))
 
def rejection_reasons(text, min_words=20, max_words=100000, prose_only=True):
    s=quality_signals(text); reasons=[]
    if not min_words <= s['words'] <= max_words: reasons.append('word_count')
    if not 2 <= s['mean_word_chars'] <= 15: reasons.append('mean_word_chars')
    if s['symbol_word_fraction'] > .1: reasons.append('symbols')
    if s['bullet_line_fraction'] > .9: reasons.append('bullets')
    if s['ellipsis_line_fraction'] > .3: reasons.append('ellipsis')
    if s['repeated_word_character_fraction'] > .5: reasons.append('repetition')
    if prose_only and s['stopword_occurrences'] < 2: reasons.append('stopwords')
    if prose_only and s['brace_line_fraction'] > .05: reasons.append('braces')
    return reasons,s
 
assert quality_signals('a b c d e a b c d e')['repeated_word_character_fraction'] == 1
assert quality_signals('a b c d e f')['repeated_word_character_fraction'] == 0
assert quality_signals('x={1:2}')['brace_line_fraction'] == 1
assert 'word_count' in rejection_reasons('')[0]
assert 'repetition' in rejection_reasons('make money fast '*30)[0]
print('quality signals',quality_signals('the river and a tree\nare in the valley'))

For language identification, lid.176 is a trained classifier for176 languages with separate full and compressed model artifacts. Flatten newlines for its API; a first1000-character truncation can miss a language switch later in the document. Sample multiple spans or score full bounded text. A model confidence is not a calibrated probability of document acceptability. No universal.65 threshold was “converged on” by C4 and RefinedWeb.

5 · Privacy, blocklists and legal review

A hostname blocklist should parse URLs rather than substring-match them, decide whether to include subdomains, normalize IDNs consistently and record list version/reason. It is not a95% coverage guarantee. A PII regex can miss international formats and non-text identifiers while falsely matching ordinary numeric strings. Presidio is a configurable detection/anonymization framework, not a proof that a corpus is private.

from urllib.parse import urlsplit
EMAIL = re.compile(r'[\w.+-]+@[\w-]+\.[\w.-]+')
PHONE = re.compile(r'\+?\d[\d ()-]{8,}\d')
 
def blocked_host(url, domains):
    host = (urlsplit(url).hostname or '').lower().rstrip('.')
    return any(host==d or host.endswith('.'+d) for d in domains)
 
def redact_demo(text):
    # Deliberately incomplete; caller must not label this privacy certification.
    return PHONE.sub('[PHONE]', EMAIL.sub('[EMAIL]',text))
 
assert blocked_host('https://sub.example.test/a',{'example.test'})
assert not blocked_host('https://notexample.test',{'example.test'})
assert redact_demo('Contact a@example.test') == 'Contact [EMAIL]'
assert redact_demo('account private-name') == 'account private-name'

Decide whether to reject, quarantine or redact a sensitive document based on policy. Redaction changes content and may create new duplicates, so hash/cluster at the appropriate stage and retain lineage. Deduplication reduces repeated exposure but a single sensitive record remains sensitive. Toxicity and quality are different labels; language/dialect bias in either must be audited. The original “regex is enough for a personal run” claim is removed, not softened into an implied guarantee.

6 · Complete optional WARC adapter and driver

This program operates only on already available local WARCs and an already available fastText language-ID model. warcio, trafilatura and fasttext are absent in the review runtime; the wrapper is unexecuted. Pure filters above are executed on authored inputs. No crawler, credentials, package install, dataset or language-model weights are fetched.

The output is a streaming NDJSON file, not an in-memory list of all survivors. A temporary output and final rename prevent a partial file from masquerading as complete; resumability still needs a manifest/cursor and replay policy, not merely the NDJSON extension. near_keep is an explicit required decision hook—use a reviewed DL049 index or a precomputed duplicate-group decision, not an unnamed hidden placeholder in the loop.

# Optional reference: local WARC/extraction/lang-ID integration; unexecuted.
def process_local_warc(warc_path, lid_model_path, output_path, usage_basis,
                       near_keep, domain_blocklist=(), min_confidence=.8):
    from pathlib import Path
    import json, hashlib, os
    import fasttext, trafilatura
    from warcio.archiveiterator import ArchiveIterator
    if not usage_basis or not callable(near_keep):
        raise ValueError('usage basis and explicit near-duplicate policy required')
    source, lid_path, output = map(Path,(warc_path,lid_model_path,output_path))
    if not source.is_file() or not lid_path.is_file() or output.exists():
        raise ValueError('local inputs required and output must be new')
    output.parent.mkdir(parents=True,exist_ok=True)
    temp = output.with_suffix(output.suffix+'.partial')
    language_model = fasttext.load_model(str(lid_path))
    counters=Counter();seen=set()
    try:
        with source.open('rb') as src, temp.open('x',encoding='utf8') as dst:
            for record in ArchiveIterator(src):
                counters['records'] += 1
                if record.rec_type != 'response' or record.http_headers is None:
                    continue
                ctype = record.http_headers.get_header('Content-Type') or ''
                if 'text/html' not in ctype.lower(): continue
                counters['html_responses'] += 1
                url = record.rec_headers.get_header('WARC-Target-URI') or ''
                if blocked_host(url,set(domain_blocklist)):
                    counters['reject_host'] += 1;continue
                html = record.content_stream().read(5_000_001)
                if len(html)>5_000_000:
                    counters['reject_oversize'] += 1;continue
                text = trafilatura.extract(html,favor_precision=True,
                    include_comments=False,include_tables=True,deduplicate=True)
                if not text:
                    counters['reject_extraction'] += 1;continue
                counters['extracted'] += 1
                labels,probs = language_model.predict(text.replace('\n',' '),k=1)
                if labels[0] != '__label__en' or float(probs[0]) < min_confidence:
                    counters['reject_language'] += 1;continue
                reasons,signals = rejection_reasons(text)
                if reasons:
                    for reason in reasons: counters['reject_'+reason] += 1
                    continue
                # Privacy hook here is demonstrative redaction, not clearance.
                redacted = redact_demo(text)
                digest = hashlib.sha256(redacted.encode()).hexdigest()
                if digest in seen:
                    counters['reject_exact_duplicate'] += 1;continue
                seen.add(digest)
                if not near_keep(digest,redacted):
                    counters['reject_near_duplicate'] += 1;continue
                item=dict(id=digest,url=url,text=redacted,usage_basis=usage_basis,
                    privacy_status='requires-policy-review',signals=signals,
                    raw_html_sha256=hashlib.sha256(html).hexdigest())
                dst.write(json.dumps(item,ensure_ascii=False)+'\n');counters['retained'] += 1
            dst.flush();os.fsync(dst.fileno())
        temp.replace(output)
    except BaseException:
        # Retain partial evidence; never publish it under the final name.
        raise
    return dict(counters)

This adapter does not filter HTTP response status codes: an HTML error response can reach extraction. A production policy must explicitly reject or separately classify non-success statuses before extraction and count them. This complete driver defines an operational boundary, not a production readiness claim: record/process isolation for hostile HTML, robust decoding, retries, privacy policy, external-sort dedup state, source-group split and manifest verification still belong in a scale-up plan. Limit concurrency by measured RAM/IO; a16-vCPU host does not imply16 unconstrained extraction workers. Include extraction failures and reject reasons in pilot reporting.

7 · Token shards that preserve IDs, tails and boundaries

A token file is not self-describing. Record little-endian dtype, token count, tokenizer digest, end-of-document ID and document offsets. uint16 holds IDs0 through65535, hence65536 possible IDs; choose dtype from the maximum ID, including specials, not merely the nominal vocabulary size. One billion two-byte IDs occupy2,000,000,000 bytes (about1.863GiB), excluding indexes/manifests.

The original writer failed to expand~, emitted at most one shard per document, dropped the final tail and could silently wrap large IDs. The bounded writer below validates IDs, emits as many shards as needed and keeps a global document-offset index across shard boundaries. An EOD separator does not itself block attention across documents. Choose document-local windows or a specified packed attention/loss policy.

from pathlib import Path
import json, hashlib
import numpy as np
 
def write_shards(documents, folder, max_id, eod, tokenizer_hash, shard_tokens=7):
    if shard_tokens <= 0 or not 0 <= eod <= max_id <= 2**32-1:
        raise ValueError('invalid shard size or ID domain')
    folder=Path(folder).expanduser();folder.mkdir(parents=True,exist_ok=True)
    if any(folder.iterdir()): raise ValueError('dedicated empty output directory required')
    dtype=np.dtype('<u2' if max_id<=65535 else '<u4')
    buffer=[];shards=[];offsets=[];cursor=0
    def flush():
        nonlocal buffer
        if not buffer:return
        name=f'shard-{len(shards):04d}.bin'; payload=np.asarray(buffer,dtype=dtype).tobytes()
        (folder/name).write_bytes(payload)
        shards.append(dict(file=name,tokens=len(buffer),bytes=len(payload),
                           sha256=hashlib.sha256(payload).hexdigest()))
        buffer=[]
    for doc_id,ids in documents:
        begin=cursor
        for token in list(ids)+[eod]:
            if not isinstance(token,(int,np.integer)) or not 0<=token<=max_id:
                raise ValueError('token outside declared range')
            buffer.append(int(token));cursor+=1
            if len(buffer)==shard_tokens:flush()
        offsets.append(dict(id=doc_id,start=begin,end=cursor))
    flush()
    manifest=dict(dtype=dtype.str,tokens=cursor,eod=eod,tokenizer_sha256=tokenizer_hash,
                  documents=offsets,shards=shards)
    (folder/'manifest.json').write_text(json.dumps(manifest,indent=2))
    return manifest
 
fixture=Path('batch7-dl-pretraining-fixtures/shards046')
# Fixed local fixture is recreated only by the uniquely scoped test harness.
if fixture.exists():
    for file in fixture.iterdir():
        if file.name=='manifest.json' or re.fullmatch(r'shard-\d{4}\.bin',file.name):
            file.unlink()
manifest=write_shards([('a',[7,4,9,2]),('b',[1,3,5,6,8,9,10,11])],fixture,
                      max_id=70000,eod=70000,tokenizer_hash='authored-byte-contract',shard_tokens=5)
reloaded=np.concatenate([np.fromfile(fixture/s['file'],dtype=manifest['dtype']) for s in manifest['shards']])
assert reloaded.tolist()==[7,4,9,2,70000,1,3,5,6,8,9,10,11,70000]
assert [s['tokens'] for s in manifest['shards']]==[5,5,4]
assert manifest['dtype']=='<u4' and manifest['documents'][1]['start']==5
print('shard counts, tail, dtype and offsets',manifest['tokens'],manifest['dtype'])

The writer bounds its token buffer by shard_tokens, but list(ids) also materializes one whole document and its offsets/shard metadata grow with the corpus. It is not a constant-memory end-to-end pipeline. Supply bounded documents; a scale-up implementation should stream IDs and store indexes externally.

The direct fixture writer is a teaching reference; on real output, stage the entire directory and publish its manifest only after validating all digests. Crash recovery must distinguish partial shards from committed ones. A long document can span multiple shards, so a DataLoader needs both shard boundaries and the document index rather than interpreting each file as a separate document.

A reload must check the manifest, not just call fromfile. This deliberately bounded validator loads the tiny fixture into memory. It requires a trusted expected tokenizer identity and checks checksums, payload sizes, contiguous document coverage and EOD placement before exposing IDs. A checksum detects accidental byte changes relative to a trusted manifest; an attacker who can replace both still needs an independently authenticated manifest to be stopped.

def validated_reload(folder, expected_tokenizer, max_id, max_tokens=10000):
    folder = Path(folder)
    m = json.loads((folder/'manifest.json').read_text())
    def require(ok, message):
        if not ok: raise ValueError(message)
    def natural(x):
        return type(x) is int and x >= 0
    require(natural(max_id) and max_id <= 2**32-1, 'ID domain')
    require(natural(max_tokens), 'token cap')
    require(m['dtype'] == ('<u2' if max_id <= 65535 else '<u4'), 'dtype')
    require(m['tokenizer_sha256'] == expected_tokenizer, 'tokenizer identity')
    require(natural(m['tokens']) and m['tokens'] <= max_tokens, 'token cap')
    require(natural(m['eod']) and m['eod'] <= max_id, 'EOD range')
    dtype = np.dtype(m['dtype']); arrays = []; total = 0
    for i, shard in enumerate(m['shards']):
        require(shard['file'] == f'shard-{i:04d}.bin', 'shard order/name')
        require(natural(shard['tokens']) and shard['tokens'] > 0, 'shard count')
        total += shard['tokens']
        require(total <= m['tokens'], 'excess shard tokens')
        expected_bytes = shard['tokens'] * dtype.itemsize
        require(shard['bytes'] == expected_bytes, 'declared bytes')
        target = folder/shard['file']
        require(not target.is_symlink(), 'symlink shard')
        require(target.stat().st_size == expected_bytes, 'payload size')
        payload = target.read_bytes()
        require(hashlib.sha256(payload).hexdigest() == shard['sha256'], 'checksum')
        ids = np.frombuffer(payload, dtype=dtype)
        require(bool(np.all(ids <= max_id)), 'stored ID range')
        arrays.append(ids)
    require(total == m['tokens'], 'total tokens')
    ids = np.concatenate(arrays) if arrays else np.empty(0, dtype=dtype)
    cursor = 0; seen_ids = set()
    for doc in m['documents']:
        require(isinstance(doc['id'], str) and doc['id'] not in seen_ids, 'document ID')
        seen_ids.add(doc['id'])
        require(natural(doc['start']) and natural(doc['end']), 'offset type')
        require(doc['start'] == cursor < doc['end'] <= total, 'document offsets')
        require(int(ids[doc['end']-1]) == m['eod'], 'document EOD')
        cursor = doc['end']
    require(cursor == total, 'document coverage')
    return ids.astype(np.int64), m
 
checked_ids, checked_manifest = validated_reload(
    fixture, 'authored-byte-contract', max_id=70000)
np.testing.assert_array_equal(checked_ids, reloaded)
assert checked_ids.dtype == np.int64

Malformed schemas may raise KeyError or TypeError before these value checks; that still rejects the manifest, but a production API should translate failures consistently. The directory must remain unchanged during verification and use: this example is not race-safe filesystem sandboxing. To obtain the first document, slice the verified global array from start to end; decide explicitly whether to retain its terminal EOD when constructing document-local windows. The toy tokenizer identity string stands in for the digest of a real frozen artifact.

8 · Shifted targets: protected boundary-safe program

A context of length T needs T+1 consecutive IDs. Input[7,4,9] predicts[4,9,2]. Do not independently shuffle inputs/targets. The preserved program below only emits windows within each document.

import numpy as np
 
def windows(documents, context):
    inputs, targets = [], []
    for ids in documents:
        for start in range(0, len(ids)-context):
            block = ids[start:start+context+1]
            inputs.append(block[:-1]); targets.append(block[1:])
    return np.array(inputs, dtype=np.int64), np.array(targets, dtype=np.int64)
 
x, y = windows([[7,4,9,2], [1,3,5,6]], 3)
assert x.shape == y.shape == (2,3)
np.testing.assert_array_equal(x[:,1:], y[:,:-1])
np.testing.assert_array_equal(y, [[4,9,2], [3,5,6]])
assert not any(2 in row and 1 in row for row in x.tolist()+y.tolist())
print('inputs', x.tolist(), 'targets', y.tolist())

Overlapping windows reuse tokens and are correlated; splitting those windows afterward leaks context. Split document/source groups first, create each split's windows separately and state whether evaluation scores each target once or repeatedly. The small protected function assumes positive context and sufficiently long documents; a production wrapper must reject context<=0 and define empty output shapes. Casting storage IDs to signed64-bit before torch embedding/loss avoids dtype incompatibility.

For packed sequences, use document segment IDs to build a block-diagonal causal mask if cross-document attention is unwanted, and mask boundary/padding targets. Aggregate the sum of valid-token losses divided by valid-token counts, not the unweighted mean of unequal batch means. Keeping EOD while allowing cross-document attention is a different training objective and must be intentional.

9 · Cost planning and modern corpus choices

Measure a bounded permitted pilot's compressed input bytes, extracted bytes, unique tokens, throughput, peak resident memory and disk amplification. Estimate required input as target_tokens / observed_tokens_per_input_byte, then test several heterogeneous shards because one shard's yield may not transfer. Estimate each stage's wall time from its own input size and sustained throughput; account for pipeline bottlenecks, temporary indexes, checkpoint/replay storage and retries. Obtain a dated provider quote only if a future run is actually planned; no3-dollar1B-token corpus or weekend runtime is promised.

Data is heterogeneous. C4 is a cleaned Common Crawl corpus; The Pile and SlimPajama/RedPajama mixtures contain multiple sources, not all simply one Common Crawl stream shrunk90–99%. RefinedWeb studies a carefully processed web corpus. FineWeb and FineWeb-Edu distinguish heuristic/dedup processing from educational scoring. DCLM benchmarks data curation with controlled model/compute tracks rather than one immutable1.4B-only setup. RedPajama-V2 exposes quality signals for downstream choices; a signal is not an acceptance decision.

FineWeb-Edu's released classifier is a frozen Snowflake-arctic-embed backbone with a trained regression head on Llama3 annotations, not a fastText model. DCLM's fastText-based filter is a distinct method with its own positive/negative construction. Nemotron-CC combines filtering and synthetic generation/rewriting methods in its reported corpus, not a guarantee that all rejected text can safely be repaired. Synthetic text can inherit generator errors and narrow style; retain generator/template/version lineage and separate human-authored evaluation. Model-collapse research does not establish a universal30/70 mix or50% maximum.

Choose a well-documented existing corpus when it meets language, domain, rights and provenance needs; choose local extraction when those requirements demand it. “Research model below10B” is not a universal cutoff. Downloadability and a permissive repository license do not eliminate document-level rights/privacy review. These choices are substantive design work even when no external-data run is performed here.

10 · Recall and complete answers

1. What is WARC and why extract it? An archival record format containing crawl responses and other record types. Filter record/MIME/status/size, extract useful text and preserve provenance; raw HTML wrappers are not the intended language-learning signal. No fixed boilerplate percentage applies.
2. What fraction survives? Measure counts and bytes by stage. The original1–2%/5M-token rule mixed units and unsupported corpus assumptions; neither an unexpectedly high nor low yield alone proves misconfiguration.
3. Why C4's brace rule, and what is the cost? It heuristically targets code-like contamination in a prose corpus; the cited implementation rejects a document on an opening brace in a qualifying line. It can reject useful technical text. The teaching policy here is explicitly different and tested.
4. Why dedup before splitting? Duplicate and near-dependent groups crossing splits distort evaluation. Low train loss may indicate repeated examples, but cannot by itself establish memorization or guarantee poor downstream results. Inspect overlap and use clean held-out groups.
5. What dtype for a32k vocabulary? If every special ID also fits, uint16 is sufficient; it represents65536 IDs. One billion IDs cost2GB payload. Larger max IDs require uint32, and the loader casts to the model's expected signed integer type.
6. Tokenizer trained before filtering? Its merge distribution may mismatch retained text. Fit on the final permitted training distribution, then freeze it. The effect must be measured; no universal30% token-budget loss follows.
7. Does a duplicate hash or PII regex prove privacy? No. A unique sensitive record remains sensitive, and regex misses many forms. Rights, privacy, toxicity, quality and duplication are separate decisions.
8. Why related sources and packed masks? A raw hash misses edited mirrors; a separator token does not reset attention. Group versions before split and specify the attention/target boundary policy.
9. Stretch runbook: on one already-permitted local WARC, compare extractor settings; log stage counts and reject samples under safe retention rules; apply reviewed language/privacy/duplicate policies; split groups; tokenize with a frozen local artifact; write/verify shard hashes, ID ranges, offsets and tails; reload and compare tokens. Report measured time/RAM/yield without comparing to a fabricated “correct”5M-token target.

Tomorrow reconstruct the funnel and list units at every arrow. After a week, deliberately inject a duplicate across candidate splits and a70000-valued special token into a uint16 plan, then explain why both must be caught before training.

Sources and optional navigation

Common Crawl overview, access guide, warcio, trafilatura API, fastText language ID, Gopher appendix, C4 tagged source, Lee et al., FineWeb, FineWeb-Edu classifier, DCLM, Nemotron-CC, RedPajama, model-collapse study.

Optional inherited context: Karpathy intro, State of GPT, RefinedWeb, RedPajama-V2. Course map · Previous DL045 · Next DL047. Former lab shortcuts toDL044 andDL054 remain optional.