Dinesh/ Blog

Subject / Published lessons

machine learning

Explore the ideas, work through an example, and save a lesson to return to.

Filters · 1 active

132 articles

Recently updated first
  1. Updated ·machine learning

    Evaluation and Red-Teaming: Protocols, Uncertainty and Safety

    Build reproducible capability, chat, domain and red-team evaluations with correct metrics, overlap analysis and uncertainty.

    20 min read
  2. Updated ·machine learning

    KV Cache Deep Dive: Correctness, Memory and Serving Budgets

    Test exact cached decoding and grouped-query attention, derive byte budgets, and distinguish paging, precision and eviction.

    20 min read
  3. Updated ·machine learning

    Quantization — INT8, INT4, GPTQ, AWQ

    Compute quantization error and metadata costs, distinguish compression algorithms and kernels, and design a complete local-artifact benchmark.

    17 min read
  4. Updated ·machine learning

    Pruning and Distillation: Exact Masks and Correct Token Losses

    Implement exact pruning masks and shifted masked distillation, derive temperature gradients, and evaluate compression tradeoffs.

    20 min read
  5. Updated ·machine learning

    Speculative Decoding — Acceptance, Residual Sampling and Cost

    Prove speculative sampling, implement bounded greedy and stochastic decoders, and compare proposal families with a measured runtime cost model.

    17 min read
  6. Updated ·machine learning

    Continuous Batching: Scheduling, Cache Admission and Capacity

    Trace static and continuous batches, enforce cache admission, derive Little’s law and diagnose latency-throughput tradeoffs.

    20 min read
  7. Updated ·machine learning

    Inside vLLM and TGI: Paging, Versioned Schedulers and Operations

    Implement a reference-counted block allocator, inspect pinned vLLM internals, and prepare honest serving and observability recipes.

    20 min read
  8. Updated ·machine learning

    Deploying an LLM API: Bounded FastAPI Streaming and Lifecycle

    Preserve SSE framing, bound admission, authenticate and clean up streaming requests, and design deployment, retry and usage contracts.

    20 min read
  9. Updated ·machine learning

    Latency vs Throughput — Little's Law, TTFT, ITL, and Load Testing

    Define honest streaming metrics, derive capacity and test SLO measurement with bounded load-test references.

    16 min read
  10. Updated ·machine learning

    Multimodal — CLIP → LLaVA

    Derive contrastive learning and visual-token interfaces, train a synthetic dual encoder and design a complete multimodal evaluation.

    15 min read
  11. Updated ·machine learning

    Agents and Tool Use

    Build a bounded tool controller with validated proposals, independent authorization, recovery and task-level evaluation.

    15 min read
  12. Updated ·machine learning

    RAG as a testable evidence pipeline, not a truth switch

    Implement chunking, exact retrieval and rank fusion, integrate evidence packing and evaluate a complete RAG pipeline.

    18 min read
  13. Updated ·machine learning

    Mixture of Experts (MoE) Intuition

    Implement sparse expert routing, derive parameter and memory accounting, and test load-balancing and dispatch assumptions.

    15 min read
  14. Updated ·machine learning

    Long Context — RoPE Scaling, YaRN, ALiBi

    Derive RoPE extension, YaRN scaling and ALiBi, then evaluate long-context systems without confusing acceptance with recall.

    15 min read
  15. Updated ·machine learning

    Red-Team + Safety Eval

    Design controlled safety tests, separate harm from refusal, and connect system controls to evidence-based release decisions.

    15 min read
  16. Updated ·machine learning

    Capstone: a reproducible model-and-evidence laboratory

    Complete a reproducible CPU laboratory or a fully specified 100M training and release project with explicit evidence gates.

    19 min read
  17. Updated ·machine learning

    Stanford CS229: Machine Learning Course

    A source-reviewed CS229 orientation with worked linear algebra, probability, optimization and runnable algorithm fixtures.

    17 min read
  18. Updated ·machine learning

    Applied LLMs — origins, evidence and applications

    A readable route through LLM foundations, attention, retrieval, prompting, audio, clinical NLP, statistical learning and deployment responsibilities.

    12 min read
  19. Updated ·machine learning

    Image classification: from a scalar model to a tested experiment contract

    Derive loss and softmax, plan controlled Fashion-MNIST experiments, and follow complete local TensorFlow references for training, callbacks and save/load.

    30 min read
  20. Updated ·machine learning

    Where LLMs Came From, and What They Actually Are

    A source-grounded history of language modelling, five core mechanisms and the boundaries between a predictor and a deployed assistant.

    22 min read
  21. Updated ·machine learning

    Reading MEDEC — detection, localization and correction are different tests

    Audit the MEDEC benchmark's data construction, exact reported scores, prompt format and metric limitations without treating a clinical NLP benchmark as deployment validation.

    18 min read
  22. Updated ·machine learning

    The ML Mental Model — Features, Labels, Train/Val/Test

    Design an honest supervised-learning experiment, diagnose five leakage mechanisms, compare losses and metrics, and test a small regression pipeline without downloads.

    24 min read
  23. Updated ·machine learning

    Bias–Variance Trade-off & Learning Curves

    Derive squared-error bias, variance and noise; measure independent refits and bootstrap limits, and interpret learning curves carefully.

    16 min read
  24. Updated ·machine learning

    Decision Trees — Gini, Entropy, Splits

    Calculate weighted impurity gains, implement a greedy tree, trace predictions and diagnose leaf-size, pruning and representation limits.

    16 min read
  25. Updated ·machine learning

    Random Forest & Bagging

    Build a per-split randomized forest, derive bootstrap coverage and covariance limits, and audit out-of-bag evaluation and feature importance.

    16 min read
  26. Updated ·machine learning

    Gradient Boosting — XGBoost, LightGBM

    Derive gradient and regularized Newton tree updates; run bounded XGBoost and LightGBM experiments with validation-controlled stages.

    16 min read
  27. Updated ·machine learning

    Model Selection — CV, Hyperparameter Tuning, Optuna

    Choose split units and time boundaries that match deployment. Distinguish nested evaluation from hyperparameter selection.

    15 min read
  28. Updated ·machine learning

    Regularization in DL — Dropout, BatchNorm, Weight Decay

    Derive dropout moments and distinguish normalization axes. Explain coupled versus decoupled decay and train-eval state.

    15 min read
  29. Updated ·machine learning

    RNNs & LSTMs — Sequences & the Vanishing Gradient

    RNNs & LSTMs — Sequences & the Vanishing Gradient: mechanisms, worked examples, assumptions and source-backed corrections with explicit experiment boundaries.

    45 min read
  30. Updated ·machine learning

    Embeddings — word2vec, GloVe, Contrastive Learning

    Embeddings — word2vec, GloVe, Contrastive Learning: mechanisms, worked examples, assumptions and source-backed corrections with explicit experiment boundaries.

    45 min read
  31. Updated ·machine learning

    Positional Encoding — Sinusoidal, Learned, RoPE

    Prove attention equivariance, derive sinusoidal and rotary position geometry, and test offsets without confusing computable positions with length generalization.

    20 min read
  32. Updated ·machine learning

    Full Transformer Architecture — Encoder + Decoder

    Assemble causal and encoder-decoder Transformers, derive residual and normalization behavior, count parameters, and train a tiny CPU character fixture.

    20 min read
  33. Updated ·machine learning

    LLM Sampling — Greedy, Beam, Top-k, Top-p, Temperature

    Derive temperature, nucleus truncation and search; implement edge-case-safe CPU sampling and translate policies into versioned API contracts.

    20 min read
  34. Updated ·machine learning

    Efficient Attention — Flash, Sparse, Linear

    Derive online softmax, distinguish exact execution from sparse and kernel operators, and verify CPU parity while specifying a separate GPU-backend proof.

    20 min read
  35. Updated ·machine learning

    Prompting — Zero-Shot, Few-Shot, Chain-of-Thought, ReAct

    Design testable prompts, bounded tool controllers and evaluation contracts; distinguish published reasoning results from offline fixtures and generated explanations.

    20 min read
  36. Updated ·machine learning

    RAG I — Chunking Strategies & Indexing

    Build a versioned RAG index with source ranges, tokenizer budgets, stale-chunk removal and access controls; test a complete local lexical-index lifecycle.

    20 min read
  37. Updated ·machine learning

    LLM Agents — Function Calling, Tools, Planning

    LLM Agents — Function Calling, Tools, Planning: mechanisms, worked examples, assumptions and source-backed corrections with explicit experiment boundaries.

    18 min read
  38. Updated ·machine learning

    LLM Evaluation — LLM-as-Judge, RAGAS, Golden Sets

    LLM Evaluation — LLM-as-Judge, RAGAS, Golden Sets: mechanisms, worked examples, assumptions and source-backed corrections with explicit experiment boundaries.

    18 min read
  39. Updated ·machine learning

    LLM Serving — KV Cache, Batching, Speculative Decoding

    LLM Serving — KV Cache, Batching, Speculative Decoding: mechanisms, worked examples, assumptions and source-backed corrections with explicit experiment boundaries.

    18 min read
  40. Updated ·machine learning

    Arrays, shapes, and the first linear model

    Master array ownership and axes, then audit attention, context and quantization claims with exact CPU checks.

    33 min read
  41. Updated ·machine learning

    Derivatives are local measurements; backprop is bookkeeping

    Rebuild the chain rule from local sensitivities through scalar autograd, matrix pullbacks and checkpointing.

    19 min read
  42. Updated ·machine learning

    Probability, cross-entropy, and stable logits

    Derive information measures and stable probability losses, then distinguish modern language and preference objectives.

    18 min read
  43. Updated ·machine learning

    Optimization: steps, momentum, AdamW, and honest experiments

    Trace optimizer updates and numerical safeguards without mistaking toy convergence for universal model performance.

    17 min read
  44. Updated ·machine learning

    Activation Functions — sigmoid, tanh, ReLU, GELU, SwiGLU, xIELU

    Derive and test activation functions, full gradient chains, gated parameter budgets and source-corrected xIELU.

    17 min read
  45. Updated ·machine learning

    Backprop by Hand: Nine Gradients and a Verified XOR Network

    Derive all nine ReLU-network gradients, preserve the tanh XOR experiment, and verify finite differences against CPU PyTorch.

    20 min read
  46. Updated ·machine learning

    Dropout and generalization — masks, scaling and controlled experiments

    Derive dropout moments, preserve leakage-resistant evaluation, test mask variants and choose regularization by controlled experiments.

    19 min read
  47. Updated ·machine learning

    Batch Norm, Layer Norm, and RMSNorm — the normalization tour

    Derive normalization axes, affine and buffer contracts, implement BN/LN/RMSNorm and test train/eval and calibration failures.

    16 min read
  48. Updated ·machine learning

    Schedulers — Warmup, Cosine, One-Cycle, and WSD

    Implement and verify warmup, linear, cosine, one-cycle and WSD clocks, resume behavior and fair schedule comparisons.

    17 min read
  49. Updated ·machine learning

    Gradient Clipping and Accumulation

    Derive weighted accumulation and global clipping, flush unequal tails and distinguish AMP, DDP and activation-checkpointing contracts.

    17 min read
  50. Updated ·machine learning

    Convolution: local shared operators, shapes and gradients

    Compute original convolution examples and general output shapes Implement multi-channel forward and backward accumulation Compare receptive fields, pooling and modern convolution designs.

    20 min read
  51. Updated ·machine learning

    CIFAR-10 training: a complete local pipeline and controlled experiments

    Build a complete local CIFAR training and selection pipeline Test normalization, parameter groups and weighted evaluation Diagnose training failures with controlled ablations.

    20 min read
  52. Updated ·machine learning

    Transfer learning: frozen features, fine-tuning and buffer discipline

    Implement local-checkpoint linear and fine-tuning regimes Freeze parameters and BatchNorm buffers deliberately Compare transfer objectives and source-target assumptions.

    20 min read
  53. Updated ·machine learning

    Data augmentation: tested mixing, task assumptions and calibration

    Implement mixup and clipped-area CutMix with exact targets Compute calibration and soft-target loss controls Choose task-valid augmentation and controlled ablations.

    20 min read
  54. Updated ·machine learning

    Vision Transformers: patch equivalence, position and attention cost

    Prove patch projection output and gradient equivalence Build and test a small complete Vision Transformer Explain position interpolation, parameter counts and quadratic cost.

    20 min read
  55. Updated ·machine learning

    Word2Vec: complete skip-gram training, geometry and honest evaluation

    Train complete skip-gram negative sampling on bundled text Derive lookup and binary-loss gradients and sampling contracts Query and evaluate embedding geometry without semantic guarantees.

    20 min read
  56. Updated ·machine learning

    RNNs from scratch: complete BPTT, character generation and structured recurrence

    Implement complete NumPy BPTT and verify every coordinate Train and generate with a complete character RNN locally Distinguish clipping, truncation, gates, attention and affine scans.

    20 min read
  57. Updated ·machine learning

    LSTM and GRU — Gates, State and Gradient Paths

    An ordinary RNN repeatedly applies a nonlinear state transformation. Its time derivative is a product of **time-varying Jacobians**, not just the recurrent matrix's eigenvalue raised to a power. LSTM introduced a separately carried cell state; the la

    20 min read
  58. Updated ·machine learning

    Seq2Seq — A Complete Length-Aware Encoder–Decoder

    A sequence-to-sequence model defines a conditional distribution over a target sequence given a source. In the fixed-state recurrent version, an encoder reads the source and passes its final state to an autoregressive decoder. For an LSTM the interfac

    20 min read
  59. Updated ·machine learning

    Why Attention — Query-Dependent Source Access

    Query-dependent access to encoder representations, recurrent attention, masking, and exact tiled attention mechanisms.

    20 min read
  60. Updated ·machine learning

    Multi-Head Attention — Shapes, Specialization and KV Budgets

    A single attention head already forms a distribution over many keys; it is not forced to pick exactly one token. Multi-head attention gives several independently parameterized distributions and value projections, then mixes their results. At fixed mo

    20 min read
  61. Updated ·machine learning

    nanoGPT Rebuilt — A Complete Tested Decoder

    The useful exercise in nanoGPT is tracing a compact decoder-only model end to end: configuration, embeddings, fused QKV attention, pre-norm blocks, final norm, tied output projection, optimizer groups and generation. Its public model/train files are

    20 min read
  62. Updated ·machine learning

    Training a Decoder on Local Text — Tiny Shakespeare Recipe

    A small character-level decoder is a useful way to inspect tokenization, shifted targets, causal loss, optimizer state and generation. Tiny Shakespeare is one possible **already-local** corpus, not data bundled or downloaded by this audit. This lesso

    20 min read
  63. Updated ·machine learning

    KV caching: correct positions, masks, memory and reuse

    Implement and test a complete cached decoder, derive cache memory, and distinguish paging, sharing, compression and eviction.

    18 min read
  64. Updated ·machine learning

    RoPE and FlashAttention: relative positions and online softmax

    Derive rotary attention and tiled normalization, test CPU parity, and separate positional extension from GPU kernel performance.

    18 min read
  65. Updated ·machine learning

    Byte-pair encoding: merges, Unicode and artifact contracts

    Build and inspect deterministic byte BPE, correct weighted merge traces, preserve Unicode bytes and freeze tokenizer artifacts.

    18 min read
  66. Updated ·machine learning

    SentencePiece and unigram: lattices, normalization and compatibility

    Derive unigram marginal likelihood and soft EM, test segmentation, and distinguish tokenizer algorithms from toolkit and normalization choices.

    17 min read
  67. Updated ·machine learning

    Pretraining data: auditable extraction, grouping and token shards

    Design a provenance-aware pretraining pipeline with tested filters, grouped splits, boundary-safe windows and validated token shards.

    20 min read
  68. Updated ·machine learning

    Chinchilla scaling: fit assumptions, budgets and serving cost

    Derive compute-constrained allocation, correct the numerical budget, and separate fixed-ratio heuristics from fitted and inference-aware scaling.

    15 min read
  69. Updated ·machine learning

    Data mixing: sampling, exposure, proxy ablations and curriculum

    Implement weighted shard sampling, measure domain exposure, run a complete synthetic ablation and explain DoReMi and token-clock schedules.

    16 min read
  70. Updated ·machine learning

    Quality and deduplication: MinHash, LSH and measured decisions

    Implement probabilistic near-duplicate retrieval with exact verification, audit learned quality policies and screen contamination with explicit limits.

    19 min read
  71. Updated ·machine learning

    Architecture choices: derive and test a 100M decoder

    Implement pre-norm, RMSNorm, RoPE, GQA and SwiGLU; reconcile exact model counts, cache budgets and modern architecture claims.

    18 min read
  72. Updated ·machine learning

    Training a 100M model: complete local recipe and budget

    Build a local-shard training state machine with exact token budgets, validation, persistence and honest resource-dependent execution limits.

    14 min read
  73. Updated ·machine learning

    Checkpointing and resuming: consistent state and durable publication

    Implement trusted local restart snapshots, test continuation and failures, and derive retention and distributed recovery boundaries.

    15 min read
  74. Updated ·machine learning

    Pretraining bugs: hypotheses, counterexamples and tests

    Diagnose stalled, nonfinite, drifting or misleading training runs through bounded interventions rather than loss-curve folklore.

    14 min read
  75. Updated ·machine learning

    Scaling from 100M to 1B — A Mental Model

    Calculate compute, memory and elapsed-time ranges across model scales, interpret published receipts, and plan infrastructure from measured constraints.

    15 min read
  76. Updated ·machine learning

    Instruction datasets: provenance, normalization and mixtures

    Audit exact dataset identities and rights, reconstruct conversation trees, and derive reproducible example and supervised-token sampling weights.

    18 min read
  77. Updated ·machine learning

    Adaptation with LoRA: low-rank updates and masked supervision

    Derive low-rank adapter gradients, inject and test a complete layer, and audit versioned SFT, merge and variant workflows.

    16 min read
  78. Updated ·machine learning

    QLoRA: NF4 storage, gradients and local integration

    Derive block-scale memory, implement a CPU NF4 teaching quantizer, and distinguish adapter gradients, paging and merge behavior.

    15 min read
  79. Updated ·machine learning

    Reward Modeling: Pairwise Likelihood, Data and Failure Modes

    Derive preference likelihoods, test reward gradients and pooling, and audit data, calibration and reward hacking.

    20 min read
  80. Updated ·machine learning

    RLHF with PPO — Intuition + Minimal Code

    Derive PPO clipping and GAE, execute a complete tiny CPU loop, and audit language-model trainer, reward and modern RL workflows.

    17 min read
  81. Updated ·machine learning

    DPO — RLHF without the RL

    Derive DPO from KL-regularized rewards, test masked log probabilities and gradients, and compare preference objectives and trainer contracts.

    16 min read
  82. Updated ·machine learning

    Google and Kaggle Gen AI study notes — the opening foundations

    An opening-reading notebook, not a completed five-day course: distinguish recurrence, attention, autoregressive generation, prompting, retrieval and fine-tuning.

    8 min read
  83. Updated ·machine learning

    Deep Learning and LLMs: an inspectable learning laboratory

    A broad deep-learning and LLM progression with a verified NumPy spine, advanced framework and systems lessons, source evidence and explicit execution boundaries.

    13 min read
  84. Updated ·machine learning

    A deep-study prompt with examples, evidence and an audio version

    Build a bounded study request, keep sources separate from narration, and evaluate generated explanations instead of mistaking fluent detail for verified knowledge.

    10 min read
  85. Updated ·machine learning

    How transformers actually attend — a weighted lookup you can calculate

    Follow queries, keys and values through a numeric attention head, test causal masking, and separate multi-head capacity from guaranteed interpretability or long-context quality.

    12 min read
  86. Updated ·machine learning

    Linear Regression from Scratch: Solvers, Geometry and Evidence

    Derive least squares, test gradients and projection geometry, recover its history, and compare stable and iterative solvers.

    20 min read
  87. Updated ·machine learning

    Logistic Regression — Sigmoid, Cross-Entropy, from Scratch

    Derive linear log-odds, stable binary cross-entropy and its gradient; separate calibration, ranking and validation-selected decisions.

    15 min read
  88. Updated ·machine learning

    Regularization — L1, L2, Elastic Net

    Derive ridge, lasso and elastic net; verify normalization, soft thresholding, fold-safe tuning and the AdamW distinction.

    15 min read
  89. Updated ·machine learning

    Evaluation Metrics — P/R/F1/ROC/PR/AUC

    Derive confusion-matrix, ROC and average-precision metrics; select thresholds without test leakage and account for prevalence and capacity.

    16 min read
  90. Updated ·machine learning

    Feature Engineering — Encoding, Scaling, Missing

    Build a fold-safe heterogeneous feature pipeline; test encoding, missingness, learned state, target cross-fitting and persistence boundaries.

    19 min read
  91. Updated ·machine learning

    Imbalanced Data — SMOTE, Class Weights, Thresholds

    Separate ranking, calibration and threshold decisions under imbalance. Apply weighted learning and fold-local resampling without leakage.

    15 min read
  92. Updated ·machine learning

    Perceptron & Activation Functions

    Derive perceptron updates and the XOR impossibility proof. Compare activation derivatives and failure modes.

    15 min read
  93. Updated ·machine learning

    Multi-Layer Perceptron — Forward Pass

    Trace forward-pass shapes and count every parameter. Explain initialization and approximation assumptions.

    15 min read
  94. Updated ·machine learning

    Backpropagation — Derived by Hand on a 2-Layer Net

    Derive affine and activation gradients including biases. Check complete NumPy gradients against numerical and autograd references.

    15 min read
  95. Updated ·machine learning

    Optimizers — SGD, Momentum, Adam, RMSprop

    Implement and compare SGD, momentum, RMSprop and Adam updates. Derive startup correction and decoupled weight decay.

    15 min read
  96. Updated ·machine learning

    PyTorch Fundamentals — Tensors, Autograd, nn.Module

    Use tensor storage, dtype, device and gradient contracts. Build registered modules and a validation-safe training loop.

    15 min read
  97. Updated ·machine learning

    CNNs — Convolution, Pooling, ImageNet Architectures

    CNNs — Convolution, Pooling, ImageNet Architectures: mechanisms, worked examples, assumptions and source-backed corrections with explicit experiment boundaries.

    45 min read
  98. Updated ·machine learning

    Transfer Learning & Fine-Tuning Classical DL

    Transfer Learning & Fine-Tuning Classical DL: mechanisms, worked examples, assumptions and source-backed corrections with explicit experiment boundaries.

    45 min read
  99. Updated ·machine learning

    Tokenization — BPE, WordPiece, SentencePiece

    Tokenization — BPE, WordPiece, SentencePiece: mechanisms, worked examples, assumptions and source-backed corrections with explicit experiment boundaries.

    45 min read
  100. Updated ·machine learning

    Attention Intuition — Sequence Bottlenecks and Soft Retrieval

    Attention Intuition — Sequence Bottlenecks and Soft Retrieval: mechanisms, worked examples, assumptions and source-backed corrections with explicit experiment boundaries.

    45 min read
  101. Updated ·machine learning

    Q/K/V Math — Scaled Dot-Product Attention Derived

    Q/K/V Math — Scaled Dot-Product Attention Derived: mechanisms, worked examples, assumptions and source-backed corrections with explicit experiment boundaries.

    45 min read
  102. Updated ·machine learning

    Multi-Head Attention — Parallel Views

    Multi-Head Attention — Parallel Views: mechanisms, worked examples, assumptions and source-backed corrections with explicit experiment boundaries.

    45 min read
  103. Updated ·machine learning

    Encoder (BERT), Decoder (GPT), Enc-Dec (T5) — When Each

    Separate Transformer families, pretraining losses and output heads; work through BERT corruption and T5 sentinels and inspect local-only inference interfaces.

    20 min read
  104. Updated ·machine learning

    Scaling Laws — Chinchilla, Compute-Optimal Training

    Derive budget-conserving language-model allocations, reconcile Chinchilla and Kaplan, and compare training-only and lifetime-cost objectives.

    20 min read
  105. Updated ·machine learning

    RAG II — Retrieval, Hybrid Search, Reranking

    RAG II — Retrieval, Hybrid Search, Reranking: mechanisms, worked examples, assumptions and source-backed corrections with explicit experiment boundaries.

    18 min read
  106. Updated ·machine learning

    Vector Databases — pgvector, HNSW, IVF

    Vector Databases — pgvector, HNSW, IVF: mechanisms, worked examples, assumptions and source-backed corrections with explicit experiment boundaries.

    18 min read
  107. Updated ·machine learning

    Multi-Agent Orchestration — LangGraph, CrewAI Patterns

    Multi-Agent Orchestration — LangGraph, CrewAI Patterns: mechanisms, worked examples, assumptions and source-backed corrections with explicit experiment boundaries.

    18 min read
  108. Updated ·machine learning

    Fine-Tuning — LoRA, QLoRA, PEFT, When NOT to Fine-Tune

    Fine-Tuning — LoRA, QLoRA, PEFT, When NOT to Fine-Tune: mechanisms, worked examples, assumptions and source-backed corrections with explicit experiment boundaries.

    18 min read
  109. Updated ·machine learning

    Multimodal LLMs — CLIP, VLMs, Audio, Video

    Multimodal LLMs — CLIP, VLMs, Audio, Video: mechanisms, worked examples, assumptions and source-backed corrections with explicit experiment boundaries.

    18 min read
  110. Updated ·machine learning

    Vectors, Matrices, and the Geometry of Data

    Derive geometry, matrix products, SVD and retrieval scores while testing the limits of each analogy.

    18 min read
  111. Updated ·machine learning

    Broadcasting — The Rule That Runs the World

    Predict shape sharing, memory costs and numerical failure cases in NumPy and transformer patterns.

    15 min read
  112. Updated ·machine learning

    One neuron: logistic regression, stable loss, and its boundary

    Train and audit a sigmoid neuron, verify exact framework parity, and prove and repair its XOR limitation.

    17 min read
  113. Updated ·machine learning

    The Forward Pass: Shapes, XOR and Parameter Counts

    Compute an MLP by hand, construct a strict-margin XOR classifier, and distinguish actual architecture counts from universal claims.

    20 min read
  114. Updated ·machine learning

    MLP on MNIST: A Complete Local-Only NumPy Recipe

    Load existing local IDX files, implement stable multiclass loss and backpropagation, and separate a synthetic contract from real digit evaluation.

    20 min read
  115. Updated ·machine learning

    Micrograd: Build and Test a Scalar Autograd Engine

    Implement complete scalar reverse-mode differentiation, shared-edge accumulation, stable cross entropy and a four-point neural network.

    20 min read
  116. Updated ·machine learning

    Vectorization: Measure Speed, Shapes and Memory

    Compare exact scalar and array programs, derive einsum contractions and FLOP counts, and measure performance without promised speedup factors.

    20 min read
  117. Updated ·machine learning

    PyTorch Tensors and Autograd: Leaves, VJPs and Safe Updates

    Port NumPy mathematics to raw PyTorch tensors and verify gradient accumulation, cotangents, graph lifetime and mutation controls.

    20 min read
  118. Updated ·machine learning

    nn.Module: Registration, Composition and Checkpoint Contracts

    Structure models with registered parameters and buffers, distinguish eval from no-grad, and verify trusted state/config round-trips.

    20 min read
  119. Updated ·machine learning

    Datasets, DataLoaders and Collate: Coverage Before Throughput

    Test indexed sampling, ragged collation, worker/rank sharding and local stream resume while treating optional cloud adapters as unexecuted.

    20 min read
  120. Updated ·machine learning

    The Training Loop, Logging, and Reproducibility

    Build and verify a resumable training state machine with RNG, sampler, optimizer, scheduler, EMA and safe checkpoints.

    16 min read
  121. Updated ·machine learning

    GPU and Mixed Precision

    Separate device placement, operation-specific mixed precision, loss scaling and measured performance without numerical myths.

    15 min read
  122. Updated ·machine learning

    Debugging — nan, inf, and Overfitting a Single Batch

    Diagnose training with exact objective contracts, finite checks, tiny-batch fits, negative controls and profiling.

    15 min read
  123. Updated ·machine learning

    Weight Initialization — Xavier, He, and μP

    Derive fan and second-moment initialization, test activation and gradient propagation, and understand bounded width-transfer parameterizations.

    15 min read
  124. Updated ·machine learning

    CNN architectures: LeNet, AlexNet, residual learning and modern backbones

    Build LeNet-like and ResNet18 models with exact parameter counts Distinguish degradation from vanishing gradients Design fair architecture and local-data comparisons.

    20 min read
  125. Updated ·machine learning

    Attention is a weighted sum with a precise mask

    Calculate attention numerically, derive its softmax backward pass, and test causal and permutation properties.

    24 min read
  126. Updated ·machine learning

    Positional Encodings — Sinusoids, Learned Tables and RoPE

    Unmasked self-attention with shared tokenwise projections and no position signal is **permutation-equivariant**: permuting input rows permutes output rows. It is not invariant unless an additional permutation-invariant readout removes row order. Thus

    20 min read
  127. Updated ·machine learning

    Transformer Blocks — Full Architectures and an Analytic Decoder

    Attention mixes information across token positions. A position-wise feed-forward network transforms features with the **same weights at each position**. Normalization sets a branch's input or output scale, and residual addition carries an explicit sk

    20 min read
  128. Updated ·machine learning

    Distributed Training Basics — DDP and FSDP

    Derive distributed-training memory and communication costs, implement rank-aware reference loops, and distinguish DDP, ZeRO and FSDP2.

    17 min read
  129. Updated ·machine learning

    Renting Your First GPU — Runpod / Modal / Vast.ai Walkthrough

    Choose GPU resources from measured workloads, compare service lifecycles, and plan safe transfer, monitoring, checkpointing and teardown.

    17 min read
  130. Updated ·machine learning

    Loss curves and evaluation: likelihood, tasks and controls

    Preserve the exact synthetic decoder experiment while restoring token-weighted metrics, downstream evaluation, uncertainty and contamination audits.

    18 min read
  131. Updated ·machine learning

    Supervised fine-tuning: objective, templates and local recipes

    Derive assistant-only supervision, test masks and conditioning gradients, and specify pinned local SFT with template and regression checks.

    16 min read
  132. Published ·machine learning

    Reading OpenAI’s Math Repository: From Research Claims to Checkable Proofs

    A deep guide to OpenAI’s math release: proof versus tests, rational approximation, zeta zero-free regions, Lean, and what formal verification does—and does not—establish.

    30 min read