Capstone: a reproducible model-and-evidence laboratory

Finish with evidence, not a graduation claim

A capstone is a coherent artifact another person can inspect and reproduce. Reaching this page does not prove you studied for eight months, trained a useful chat model or belong to a percentile of practitioners. The achievement is whatever your saved artifacts demonstrate. A failed well-designed experiment is more useful than a fabricated success table.

There are two legitimate routes here: a bounded CPU model-and-evidence laboratory, and a larger 100M pretraining/instruction/release project. The first remains a complete learning deliverable; it does not silently replace the second. The larger route has explicit resource, license, evaluation and publication gates. Uploads, GPU rental and live deployment require a separate decision about permissions, budget and release readiness.

Prerequisites are the mathematical/training checks, tokenizer/data pipeline, checkpointing, SFT/preference objectives, inference budgeting and safety evaluation from the preceding lessons. Inventory what you actually completed. Budget storage from the chosen model/optimizer/checkpoint count, not a mandatory50GB figure; estimate time from a measured bounded run rather than “two afternoons.”

CPU route: the deliverable and its boundaries

Build a local folder containing the NumPy decoder, tokenizer experiment, fixed synthetic training task, evaluation report, cache-parity test and local retrieval baseline. Keep the synthetic model and document assistant separate: a five-symbol decoder is not a natural-language grounded generator. Record source/config/data hashes, environment versions, exact commands and raw outputs.

CPU milestone 1: specify a question you can answer

Use the repeated-cycle task from the training and evaluation lesson or another generated task with a known structure. State vocabulary, context limit, target mask and train/validation/test grouping. Group rotations of the same cycle before splitting to prevent leakage; save actual split IDs, not only a seed. Declare the optimization/trial budget before reading test results.

Compare a uniform distribution and a count-smoothed bigram baseline with the neural model. A tiny model need not beat them on every dataset. Report misses and investigate whether later positions improve over first-position uncertainty. Do not select thresholds or architectures on the final test set.

CPU milestone 2: prove the objective and evaluate training

Pass all-parameter finite differences on a small deterministic configuration, including repeated embedding IDs, batch size one, multiple sequences and several context lengths. Test future-token perturbation: earlier logits must not change. Check the unique parameter count and every loss reduction/normalization axis against hand calculations.

Then train the full 688-parameter reference model from the NumPy decoder section, saved as tiny_transformer.py beside the training script. Report initial/final training NLL, validation/test NLL, perplexity, valid-token counts, baseline values and updates. The192-parameter gradient fixture is a smaller correctness audit, not the same trained model. Save early/late-position losses and every attempted experiment. A printed value copied from a preceding lesson is not your run's result.

CPU milestone 3: resume and reproduce inference

Save architecture, explicit numeric arrays, optimizer moments, RNG state, update counter and tokenizer/split hashes. The prior training lesson's in-memory recovery is not yet fresh-process disk recovery. Add disk persistence with documented dtype/shape validation, reload in a fresh process and compare logits plus the next update under the same controlled environment.

Use the retained NumPy cache test as a starting point and compare full-prefix and cached logits at each prefix up to the limit. Test overflow, changed prompt, model/adapters and position configuration. Calculate KV bytes independently of weights/activations. If timing the CPU teaching implementation, name machine, workload and repetitions; do not extrapolate to a GPU serving benchmark.

CPU milestone 4: evaluate retrieval independently

Create at least twenty authored questions over authorized small documents: direct questions, paraphrases, ambiguous/no-answer queries and permission-denied cases. Record gold supporting IDs and user access sets. Test the lexical/extractive baseline before any neural generator and report retrieval misses, unsupported claims and abstention separately.

Use inert adversarial text whose instruction conflicts with the task. Document text must not grant access, execute code or approve a write. Passing a fixture is a regression check, not injection immunity. If adding a pretrained generator later, identify its exact checkpoint and assess its separate contributions and risks.

Join the final lessons through one evidence contract. Keep a manifest row for each evaluated configuration: model/checkpoint and tokenizer hashes; trained, configured and actually tested lengths; positional method/base/scaling and rotary layout; cache dtype, KV heads and retained-token policy; retrieval/index/chunk versions; authenticated access set; evaluator and case-set hashes. The synthetic five-symbol decoder and the document assistant need separate configuration IDs. A passing hash manifest proves identity, not model quality or coverage of all preceding lessons.

For each answerable question, save gold authorized support IDs, retrieved IDs, packed IDs after the input budget, actual token count, evidence position, answer, claim-support labels and raw outcome. This yields a useful failure trace: retrieval miss → packing loss → present-but-unused evidence → unsupported answer. Do not let a summary or a retrieved document silently stand in for the gold support. For no-answer and permission-denied questions, record abstention/denial separately; there is no nonempty gold support set from which to compute ordinary recall. Evaluate benign questions as well as inert conflicting instructions, keeping model proposals and controller effects in different fields.

A hand-worked four-case report makes these distinctions concrete. Cases A and B are answerable; C has no answer; D requests a denied document. Suppose retrieval finds A's support but misses B's; packing then drops A's support. Retrieval support recall is 1/2, packed support recall is 0/2. If both answerable cases abstain, answerable correctness is 0/2, even if C correctly abstains and D is denied. Calling this “50% accurate” hides which task failed. If A instead returns the gold answer without packed support, record answer correctness 1/2 but grounded correctness 0/2; the answer may be a lucky guess or leakage. These are authored arithmetic cases, not model results.

Cross this report with the long-context lesson: start/middle/end evidence, longer distractors, a short-context control, absent-evidence negative control and multi-hop/aggregation cases. For the larger design, moving from 2,048 to 8,192 retained tokens at BF16 changes the conventional K/V payload from 24 MiB to 96 MiB per sequence (12 layers, 4 KV heads, width 64). Dense score elements grow sixteenfold for a fourfold length increase. Neither arithmetic result certifies an 8k positional configuration or useful retrieval: require boundary/overflow tests and full-prefix versus cached logits under the same fixed configuration. The CPU route can exercise its own much smaller supported lengths; it must not claim to have trained or evaluated the 100M model.

Finally join the safety lesson: retain matched clean/inert-conflict cases, trusted cross-owner denial plus allowed-owner success, independent harm/refusal/utility labels and incomplete-run counts. If retrieval quality improves while permission checks regress, the candidate is rejected. If memory fits but middle-position quality fails the declared task bar, shorten the supported length or change retrieval/packing and rerun. Preserve failed candidates instead of averaging away a release blocker. This is a concrete capstone integration gate, not approval of the whole curriculum, an unrun model or a public deployment.

A rubric with observable evidence

An educational pass suggestion is80/100 and no unresolved target leakage, wrong-gradient defect, unauthorized data use or invented result. It is a chosen rubric, not an external research standard. A failed model can earn protocol/analysis points; fabricated success earns none.

A minimal result record

This exact retained program checks illustrative arithmetic, not a trained model result.

import math
 
report = {'task': 'schema demonstration only', 'token_nll_sum': 26.,
          'valid_tokens': 10, 'test_used_for_tuning': False,
          'limitations': ['illustrative arithmetic, not a model benchmark']}
assert report['valid_tokens'] > 0
assert not report['test_used_for_tuning']
mean_nll = report['token_nll_sum'] / report['valid_tokens']
assert abs(mean_nll-2.6) < 1e-12
print('example schema NLL', mean_nll, 'PPL', math.exp(mean_nll))

Attach a checkpoint digest, evaluator digest, data/split digest, decoding settings, counts and raw-run path to a real result record. Recompute a score from its saved numerator/denominator. Bind every model-card number to the same checkpoint offered to readers; similar loss curves do not justify carrying forward a prior checkpoint's evaluations.

Larger route: freeze a real 100M architecture

The original d768/L12/GQA4/vocab32000/SwiGLU3072 configuration does not have100M parameters. A three-matrix SwiGLU differs from a two-matrix4d GELU FFN. Use d768,12layers,12query heads,4KV heads, head width64, SwiGLU intermediate2048, tied embeddings, bias-free projections and RMSNorm to obtain 100,092,672 unique parameters. Context2048 and RoPE base10000 are architectural choices, not proof of later8k capability.

V,d,L,H,K,f=32000,768,12,12,4,2048
head=d//H
embedding=V*d
qkv=d*(d+2*K*head)
output=d*d
ffn=3*d*f
norms=2*d
count=embedding+L*(qkv+output+ffn+norms)+d
assert count==100092672
old_count=embedding+L*(qkv+output+3*d*3072+norms)+d
assert old_count==128404224
# One billion supervised targets with global 16 sequences x 2048 targets/update.
tokens=10**9; batch_tokens=16*2048
updates=(tokens+batch_tokens-1)//batch_tokens
assert updates==30518 and tokens%batch_tokens==18944
print('100M design, old design, updates and final targets',count,old_count,updates,tokens%batch_tokens)

A one-billion-token run is a proposed budget, not a guarantee of coherence or useful chat. With microbatch2, accumulation8 and one worker, global sequences/update=16 and supervised targets/update=32768 when every2048-target window is valid. There are30517 full updates plus18944 targets in a final partial update. Mask the remainder exactly; do not claim100000steps with unspecified “batch1024 and accumulation8” equals1B tokens. Estimate wall time as supervised tokens/measured target-token throughput plus evaluation/checkpoint overhead; keep cost assumptions dated.

Complete local training procedure: data to restartable checkpoint

The larger procedure is unexecuted here. It specifies the artifacts and operations needed, rather than commands for nonexistent prepare_data.py/sft.py wrappers.

1. Data manifest. Enumerate locally authorized sources with license, consent/privacy policy, retrieval/version, raw hash, extraction/filter rules and document IDs. Normalize and deduplicate before document-grouped splitting. Record rejected counts. Train a tokenizer on training text only, save tokenizer/vocabulary/normalization/special-token artifacts and test encode/decode boundary behavior.
2. Shards. Tokenize each split deterministically, choose a dtype that can represent every ID, preserve document boundaries/offsets and write shard hashes. Define EOS/boundary and packed-attention semantics. Validate shard tails and final-window sampling; a dataset name is not a reproducible dataset identity.
3. Model. Instantiate the exact architecture above using the fully developed decoder from architecture choices, or an explicitly compatible tested library configuration. Run tiny causal/shape/gradient tests before scaling. Check tied parameter identity and optimizer groups; tied embeddings must not be decayed/updated twice.
4. Optimization. Use shifted next-token CE with a declared valid-target denominator. Declare optimizer, learning-rate schedule endpoints, clipping, microbatch, accumulation, dtype and total target tokens. Scale loss correctly across accumulation and distributed workers; a partial last update must use its actual supervised count. No external experiment tracker is required: local structured metrics and hashes suffice.
5. Monitor and evaluate. Save token clocks, learning rate, gradient norm, train/validation NLL, throughput, memory and fixed generation probes. Uniform/bigram baselines and a tiny overfit test diagnose failures before expensive runs. Evaluate held-out tokens with summed NLL/valid-token count; compare perplexity only under the same tokenizer/data/masking.
6. Checkpoint. At an optimizer boundary save model, optimizer, scheduler, RNG, sampler cursor, token/update clocks and manifests. Publish local checkpoint files atomically with integrity checks. Retain a previous verified checkpoint. Interrupt a small pilot, reload in a fresh process and compare its next update before trusting a long run.

The existing pretraining recipe and checkpointing lesson provide complete implementations. The capstone's responsibility is to freeze their exact chosen versions and integration contract, not copy an unverified command string. A100M parameter target and real corpus remain unexecuted until their own manifests, logs and checkpoints exist.

SFT and optional preference adaptation

Prepare licensed instruction records with explicit system/user/assistant roles and group conversations across splits. A suggested20K mixture is a design choice; Dolly/OA/domain/safety records have distinct licenses and provenance, and an OA branch must be reconstructed by IDs rather than flattening unrelated messages. Quality-score differences are not automatically valid pairwise preferences. Inspect domain/privacy contamination and rights before using any source.

Use the chosen model/tokenizer's chat template; ChatML is not universal. Add any special tokens deliberately, resize embeddings when needed and save their IDs/template. Mask system/user/padding/visual inputs if using assistant-only loss, include intended answer end tokens and shift once. Print tokens, labels and target counts for one example. Packing must preserve the intended cross-example attention boundary, not merely concatenate masked labels.

Run a tiny overfit test and then a declared SFT budget. Compare base/SFT on task utility, text-only regressions, refusal/over-refusal and held-out token loss. LoRA can reduce trainable state but does not guarantee80%quality at5%memory; activations/base weights remain. Declare adapted modules/rank/scaling/dropout and whether deployment merges or loads adapters.

For DPO, save same-prompt chosen/rejected pairs with an explicit preference rubric and a frozen reference checkpoint. The objective uses the policy-versus-reference log-probability-ratio difference for chosen versus rejected responses, multiplied by beta inside a negative log-sigmoid. In the underlying KL-regularized formulation, beta controls the regularization relationship; “larger beta always means a more aggressive preference push” is false. Its finite-step gradient effects are not a monotone quality dial. Hold tokenization, masks and reference identity fixed; report chosen/rejected margins and evaluation utility, not only training preference accuracy.

A pilot might compare base, SFT and SFT+DPO under the same30authored questions and safety controls. Thirty questions are a development diagnostic, not MT-Bench. MT-Bench has its own prompts, multi-turn protocol and judge settings; do not label a human single-turn custom score as that benchmark. Preserve a final held-out set and failed outputs.

Compression, export and serving acceptance

Before quantization, export a locally reloadable model with config, tokenizer and safe tensor weights. Check missing/unexpected keys, tied weights and logits against the training checkpoint. A custom decoder does not become AutoModelForCausalLM-compatible merely by naming a directory; either implement/register the correct configuration/model mapping or convert into an actually compatible supported architecture with verified tensor mapping. Prefer reviewed local code over asking readers to enable arbitrary remote code.

For an already provisioned, pinned llama.cpp checkout supporting the chosen architecture, the reference conversion sequence is convert_hf_to_gguf.py followed by llama-quantize with a selected scheme such asQ4_K_M. Tool paths/build names vary by revision; inspect that checkout's help. No compiler/dependency install/conversion is run here. Preserve a high-precision GGUF baseline, quantize a separate artifact, then compare token IDs, special tokens, prompt template, logits where comparable, held-out NLL and fixed generated outputs. A100M four-bit weight payload starts around50MB, but scales, metadata and higher-precision tensors mean file size is not automatically60MB or phone compatibility.

A complete acceptance matrix covers library export, high-precision GGUF and quantized GGUF: load without network, run the same five representative prompts (including failure/benign safety controls), compare task metrics and measured memory/latency on named hardware. Five prompts catch wiring errors, not80%of all quantization bugs. Retain original/quantized hashes and distinct scores. Serving additionally needs bounded inputs/output, auth, timeouts, cancellation, queue limits, health checks and rollback. A local CLI generation is not a production-service test.

Local inference and the fresh-environment contract

The following unexecuted reference assumes a supported local Transformers4.57.1 model and saved chat template. It deliberately uses local-only loading and does not silently download a different artifact.

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
path='output/release-model'
tokenizer=AutoTokenizer.from_pretrained(path,local_files_only=True,trust_remote_code=False)
model=AutoModelForCausalLM.from_pretrained(path,local_files_only=True,trust_remote_code=False)
model.eval()
inputs=tokenizer.apply_chat_template([{'role':'user','content':'Explain one limitation of this model.'}],
    tokenize=True,add_generation_prompt=True,return_tensors='pt',return_dict=True)
with torch.inference_mode():
    output=model.generate(**inputs,max_new_tokens=64,do_sample=False)
print(tokenizer.decode(output[0,inputs['input_ids'].shape[1]:],skip_special_tokens=True))

A “three-line install” is shorthand for testing the reader's path in a clean supported environment, not a literal no-dependencies guarantee. Provide exact supported Python/library versions, artifact hashes, device/memory expectations, an example command and expected output shape. Test local offline loading before a remote download path. If an inference wrapper is packaged, test its build/install in an isolated permitted environment and inspect included files; that packaging/release work belongs to a separately authorized owner, not this audit.

Publication procedure and model card

Release is optional until safety, licensing, privacy and artifact gates pass. Use the hosting service's supported login/masked-credential flow; never paste a write token into a public post, shell script or model card. Create a private staging repository only after explicit authorization, upload an allowlisted bundle, inspect it, and make it public only after the owner approves. Avoid git add . in a training directory: optimizer dumps, raw data, secrets and private logs can leak.

The documented Hugging Face HfApi.upload_folder supports selecting files with allow_patterns/ignore_patterns. The publishable bundle should contain approved weights, model config, tokenizer/template, optional tested GGUF, model card, example inference code, license notices and a checksum manifest—not raw training data by default. Download/reload the exact remote revision in a fresh permitted environment and verify hashes before announcing it. Upload success alone does not establish reader usability.

Write the card during training, not from memory at the end:

• Identity: release/checkpoint and code hashes, architecture/count, tokenizer/template, base/adapters if any, version/date.
• Intended and excluded uses: local learning and explicitly tested domains; no unsupported medical/legal, factual or production claims.
• Data: exact sources/versions, rights, preprocessing, split grouping, supervised tokens and limitations. Choose a license based on actual rights; do not copyApache2.0 blindly.
• Training: optimizer/schedule, dtype, batch/accumulation, updates/tokens, hardware and measured resource use. Mark missing stages “not run.”
• Evaluation: dataset/evaluator hashes, decoding settings, counts, metric definitions, uncertainty, strata, baselines and failures. Avoid sample numbers such asPPL12.3 or71%refusal presented as real.
• Reproduction: local and approved remote inference path, environment, integrity checks and known compatibility constraints.
• Risks and operations: known failure modes, safety tests not performed, reporting contact, support expectations and rollback/version policy.

Model cards communicate evidence boundaries; they are not the only possible channel (interfaces and access controls matter too), and documentation cannot make an unsafe release acceptable. Compare versions on a stable evaluation set; if the evaluator changes, rerun both or label results incomparable.

Writeup, community and the next project

A useful post has a concrete measured hook, personal motivation, architecture diagram, three important decisions, one failure investigation, an evaluation table with bad as well as good outputs, a reproducible usage example and a next-step hypothesis. Around1500words and3–5code snippets are optional editorial targets, not proof of quality; use as many clear figures as the argument needs. Quote real commit messages only with permission and when they are informative. Do not invent an eight-month journey or claim frontier performance from a tiny model.

Choose a canonical personal post/model card and share to relevant communities only after the release gate and explicit publication approval. Hugging Face forums, EleutherAI discussions, LocalLLaMA, technical newsletters and paper feeds can provide feedback, but no Tuesday9am posting schedule guarantees reach. An internal/private artifact can be the right choice for sensitive data or unfinished risk controls; it is not inherently “learning slowly.”

Three bounded follow-on proposals:

1. Distill100M→25M. Fix a task and teacher/student tokenizer policy, train against hard targets plus declared soft-target loss, compare quality/memory/latency on the same device. Start with a two-week feasibility milestone; decide a longer schedule from results, not a promised three-month success.
2. Tool-use adaptation. Use synthetic typed tool-call/observation trajectories with held-out branches, retain the controller permission boundary and compare against a fixed pipeline. A first milestone is validated arguments and safe partial outcomes, not a public autonomous bot.
3. Multimodal interface. Start with a frozen local vision tower, projector and decoder, test the image-shuffle control, then visual instruction tuning if justified. Measure text-only regression and patch-budget effects before proposing a six-week deliverable.

A domain-fine-tuning/service extension can use SQL/Spark/data-engineering tasks with sandbox execution and user-scoped data. Recruit testers only with informed scope and a rollback path. Pick one project based on the diagnosed bottleneck; distributed100+GPU training, scaling-law fitting, state-space/MoE frontiers and product discovery remain separate skills. A model config predicts storage/arithmetic under assumptions, not its quality.

Oral defense and worked recall

1. Four release artifacts? Weights in a supported format, optional independently tested quantized weights, an honest model card and runnable inference example. Add config/tokenizer/license/hash manifest so those four can actually work.
2. Fresh-install test? A new supported environment loads the exact release and produces output through the documented path. It tests packaging/compatibility, not factual quality or safety; those are additional gates.
3. Most important card section? No single section substitutes for the bundle. Known limitations tell users where not to rely on it; provenance/results/reproduction make those claims inspectable.
4. Trustworthy writing technique? Show one failed output and its evaluation context alongside successes; use the uploaded checkpoint's current results, not an older best run.
5. Transformer block and shapes? Pre-norm attention plus residual, then pre-norm feed-forward plus residual. Q has (B,H,T,d_h); K/V may have fewer KV heads under GQA, expanded/grouped by the defined mapping. Causal softmax(QK.T/sqrt(d_h))V returns head outputs, concatenated/projected to (B,T,d).
6. Why mask if labels are shifted? Earlier input positions could otherwise attend to a later token containing their target. Shifting labels does not enforce causal information flow.
7. Repeated embedding IDs in gradient tests? They require several uses to accumulate into the same parameter row. A scatter-overwrite bug can pass a unique-ID fixture but fail this one.
8. Why isn't RAG a fix for every low score? It supplies external evidence; it does not repair wrong gradients, poor reasoning, invalid permissions or automatically ensure support.
9. What justifies a larger model? A diagnosed bottleneck, authorized data, a bounded measured hypothesis and better held-out outcomes—not a round parameter count.
10. One-paragraph project proposal? “Distill the current locally verified decoder into a smaller student for the same held-out synthetic/domain task. Freeze the evaluator and tokenizer policy; compare hard-only versus soft-target loss, report quality/latency/memory and stop if student quality falls below the predeclared bar. No public release or new compute purchase is part of this first milestone.”

For spaced review, revisit arrays/outer products, causal attention, tokenizer/data splits, checkpointing, SFT/DPO, cache/serving and the six final advanced topics. Implement one operation from memory and compare against an oracle. Completion is another reader reproducing your stated experiment and understanding its limits—not an automatic instruction to publish today.

Sources and return path

Model cards paper, Hugging Face cards, Hub upload guide, DPO, Transformers4.57.1 chat templates, llama.cpp. Local CPU fixture checks do not certify100Mtraining, SFT/DPO, export, quantization, remote installation or deployment; those remain unexecuted project stages here.

Previous: safety evaluation · Back to the complete course map

Optional reading

Explore these supplementary resources after the exercises. Video titles and availability have not been independently verified; use the primary sources above for technical claims.

• Let's build GPT: from scratch, in code, spelled out
• Intro to Large Language Models
• Hugging Face Model Cards
• A Recipe for Training Neural Networks