Stanford CS229: Machine Learning Course

What this CS229 guide covers

These notes began as a study journal: a first-lecture link, a prerequisite map and a plan to work through the rest of Stanford CS229. This revised page is a source-reviewed orientation with bounded worked prerequisite examples, not a claim that every lecture has a separate finished post or that the author completed the course. The roadmap below remains a study route with concrete derivations and implementation tasks; it is not silently erased because those tasks take time.

Use the official Autumn2018 syllabus to identify the linked Andrew Ng-era lecture order, and the official consolidated notes for technical derivations. The latter snapshot identifies Tengyu Ma and Andrew Ng and is dated23August2026; it is not the identical2018 handout. The old roadmap incorrectly placed neural networks/RL at particular lecture numbers; editions differ. GDA means Gaussian Discriminant Analysis, not “Generative Discriminative Analysis.”

The original Lecture 1 video and archived introduction slides are optional orientation resources. The official linear-algebra review supplies prerequisite notation. A source link is not evidence that every statement in these personal notes came from that particular video minute. Preserve the original shared Lecture 1 URL and repository linear-algebra PDF pointer as optional historical resources alongside the official review. Their exact destinations are retained without claiming a fresh video watch, availability check, or equivalence between the repository PDF and today's official document.

The learning problem and notation

Supervised learning observes labelled pairs (x_i,y_i): regression predicts a quantity; classification predicts one of a declared set of labels, for example0 through K−1 for K classes. Unsupervised learning uses unlabelled observations to estimate density, clusters or representations. Reinforcement learning learns decisions from interaction/rewards under a state/action/transition model. These are useful categories, not an exhaustive taxonomy: self-supervision constructs targets from data, and semi-supervised learning mixes labelled/unlabelled data.

Let m be examples, n features, X an m×n matrix with examples as rows, theta an n-vector and predictions Xtheta. An intercept can be added as a column of ones, increasing parameter dimension. Superscript (i) labels an example, not exponentiation. Write shapes beside every equation; many “calculus bugs” are silent broadcasting bugs.

Linear algebra: geometry before library calls

For u=(3,4), L1norm=7,L2norm=5,L-infinitynorm=4. Inner products measure projections only relative to the chosen geometry; cosine requires nonzero normalized vectors. Matrix multiplication composes linear maps, is generally noncommutative and requires inner dimensions to agree. Trace is the sum of a square matrix's diagonal; tr(AB)=tr(BA) when dimensions make both products valid, but that does not mean AB=BA.

A linear system Ax=b is consistent when b lies in A's column space. A square invertible matrix has a unique solution; a rank-deficient system may have none or infinitely many. Gaussian elimination/LU solves suitable systems without explicitly constructing A inverse. For least squares use QR/SVD or lstsq; normal equations square the condition number and can be numerically fragile.

Eigenvectors satisfy Av=lambda v. Real symmetric A admits an orthonormal eigenbasis; arbitrary matrices need not have real eigenvalues or a full eigenbasis. SVD writes A=U Sigma V.T with nonnegative singular values and applies to rectangular matrices. Zero singular values expose rank deficiency; truncating to the largest singular values gives the best low-rank approximation under standard spectral/Frobenius norms. For symmetric positive-definite A, all eigenvalues are positive and x.T A x>0 for nonzero x.

A positive-definite Hessian at a stationary point gives a strict local minimum under appropriate smoothness. It does not establish global convexity from one point. For a twice-differentiable function on a convex domain, a positive-semidefinite Hessian throughout establishes convexity. Convexity does not guarantee a unique minimizer or that an arbitrary gradient step converges.

Probability, likelihood and regularization

Conditional probability requires a nonzero conditioning probability. Bayes' rule combines likelihood and prior; a common diagnostic trap ignores prevalence. If prevalence=.01,sensitivity=.9,false-positive rate=.05, then positive predictive value is .009/(.009+.0495)≈.153846, not90%. A density can exceed 1 while its integral is 1; a distribution with a density assigns zero probability to any single point. For example, a uniform density on [0, 0.5] is 2 there but integrates to 1.

For a sample space Omega, a probability measure is nonnegative, has P(Omega)=1 and is additive over disjoint events (countably additive in general). A PMF gives masses at discrete values; a PDF must be integrated over a range. The CDF is F(t)=P(X≤t) in either case. Variance is E[X²]−E[X]² when finite. These definitions specify what a likelihood model is allowed to mean.

A Bernoulli variable has mean p and variance p(1−p); a Binomial counts Bernoulli trials under its assumptions; Poisson models counts with its own independence/rate assumptions; Gaussian, exponential and uniform models describe different supports/tails. The exponential's memorylessness is an assumption, not a generic waiting-time fact. A multivariate Gaussian has mean vector and positive-semidefinite covariance; a nonsingular density needs positive-definite covariance. Expected value is linear without independence; variance of a sum includes covariance terms.

MLE maximizes data likelihood. With independent Gaussian regression noise of fixed variance, maximizing likelihood is equivalent to minimizing squared residuals. With independent Bernoulli outputs whose probability is the sigmoid of a linear predictor, it yields logistic cross-entropy. MAP adds a prior: a zero-mean Gaussian prior on coefficients produces a squared-parameter penalty in the negative log posterior, with scaling tied to noise/prior variances and the objective's sum/mean convention. It is not equivalent to every arbitrary regularizer; priors and parameterization matter.

Entropy H(p)=-sum p log p, cross-entropy H(p,q)=-sum p log q and KL sum p log(p/q) obey H(p,q)=H(p)+KL(p||q) when defined. KL is nonnegative, not symmetric and not a distance metric. If q assigns zero mass where p is positive, KL is infinite. These definitions explain why predictive log loss punishes confident wrong answers.

Calculus and optimization through least squares

Take J(theta)=||Xtheta-y||²/(2m). A perturbation dtheta changes residuals by Xdtheta; expanding the square gives gradient X.T@(Xtheta-y)/m and Hessian X.T@X/m. The Hessian is PSD because v.T H v=||Xv||²/m. It is PD iff X has full column rank. At a stationary point, normal equations are X.T X theta=X.T y; do not invert the matrix merely because the equation is written that way.

A partial derivative varies one coordinate while holding the others fixed. For a vector-valued function f from n inputs to k outputs, its Jacobian has shape k×n; a scalar function's gradient collects its n partial derivatives, and its Hessian is n×n. For f(g(x)), multiply Jacobians in composition order: J_f(g(x)) J_g(x). This is the chain rule used by the two-layer derivative fixture, not elementwise multiplication of arbitrary matrices.

First-order Taylor expansion gives J(theta+delta)≈J+g.T delta; under the Euclidean norm the negative gradient is locally steepest descent. Under another metric/preconditioner that direction changes. Second-order expansion adds .5*delta.T H delta, motivating Newton's equation H delta=-g. Solve this linear system; use damping/line search when curvature or step quality is problematic. Local quadratic convergence requires regularity and a sufficiently close starting point, not just the word “Newton.”

For a convex quadratic with largest Hessian eigenvalue L > 0, gradient descent with step strictly between 0 and 2/L contracts nonflat eigen-directions. Too large a step can diverge even for a convex objective. A flat minimum set demonstrates nonuniqueness. SGD uses a sampled example/minibatch estimator under a specified sampling scheme; noise, schedules and assumptions determine convergence.

Lagrange multipliers enforce equality constraints through a stationarity system. Inequality constraints add nonnegative multipliers and complementarity (KKT conditions). A simple example is minimizing (w1²+w2²)/2 subject to w1+w2=2: symmetry/stationarity yields w=(1,1), not an unconstrained minimum at zero. SVM dual variables arise from these constrained optimality conditions.

Runnable prerequisite and supervised-learning laboratory

All examples use generated arrays on the existing CPU NumPy runtime. They are correctness/shape fixtures, not real-dataset model performance. The program gives least squares, finite differences, Newton/ridge, locally weighted regression, logistic regression, GDA and Bernoulli Naive Bayes with checks.

import numpy as np
np.set_printoptions(precision=6)
X=np.array([[1.,-1.],[1.,0.],[1.,1.],[1.,2.]])
y=np.array([-1.,1.,3.,5.])
m=len(X)
theta=np.linalg.lstsq(X,y,rcond=None)[0]
np.testing.assert_allclose(theta,[1,2],atol=1e-12)
H=X.T@X/m
cost=lambda w: np.square(X@w-y).mean()/2
w=np.array([.2,-.3]);g=X.T@(X@w-y)/m
eps=1e-6
fd=np.array([(cost(w+eps*np.eye(2)[j])-cost(w-eps*np.eye(2)[j]))/(2*eps) for j in range(2)])
np.testing.assert_allclose(g,fd,rtol=1e-8,atol=1e-8)
np.testing.assert_allclose(w-np.linalg.solve(H,g),theta,atol=1e-12)
step=1/np.linalg.eigvalsh(H).max()
wgd=np.zeros(2)
for _ in range(200): wgd-=step*X.T@(X@wgd-y)/m
np.testing.assert_allclose(wgd,theta,atol=1e-10)
# Convex but nonunique: duplicated columns.
rank_def=np.array([[1.,1.],[2.,2.]])
assert np.linalg.matrix_rank(rank_def)==1
np.testing.assert_allclose(rank_def@np.array([1.,0.]),rank_def@np.array([0.,1.]))
U,S,Vt=np.linalg.svd(X,full_matrices=False)
np.testing.assert_allclose(U@np.diag(S)@Vt,X,atol=1e-12)
assert np.isclose(.01*.9/(.01*.9+.99*.05),2/13)
# Ridge leaves intercept unpenalized under this explicit convention.
ridge=np.linalg.solve(X.T@X+np.diag([0.,1.]),X.T@y)
assert abs(ridge[1])<abs(theta[1])
 
def local_linear(X,y,query,tau):
    if tau<=0: raise ValueError('bandwidth')
    weights=np.exp(-np.square(X[:,1:]-query[1:]).sum(1)/(2*tau*tau))
    return query@np.linalg.lstsq(X*np.sqrt(weights[:,None]),y*np.sqrt(weights),rcond=None)[0]
assert np.isclose(local_linear(X,y,np.array([1.,.5]),1.),2.)
 
def sigmoid(z): return np.exp(-np.logaddexp(0.,-z))
labels=np.array([0.,0.,1.,1.])
w=np.zeros(2)
logloss=lambda v: np.mean(np.logaddexp(0,X@v)-labels*(X@v))
initial=logloss(w)
for _ in range(200): w-=.2*X.T@(sigmoid(X@w)-labels)/m
assert logloss(w)<initial and np.array_equal(sigmoid(X@w)>.5,labels.astype(bool))
# Shared-covariance Gaussian discriminant model on a nonsingular fixture.
a=np.array([[-2.,0.],[-1.,1.],[-1.,-1.],[1.,0.],[2.,1.],[2.,-1.]])
b=np.array([0,0,0,1,1,1])
means=np.stack([a[b==j].mean(0) for j in (0,1)])
centered=a-means[b]
cov=centered.T@centered/len(a)+1e-6*np.eye(2)
prior=np.bincount(b)/len(b)
scores=np.stack([-.5*np.sum((a-mu)*np.linalg.solve(cov,(a-mu).T).T,axis=1)+np.log(prior[j])
                 for j,mu in enumerate(means)],axis=1)
assert np.array_equal(scores.argmax(1),b)
# Bernoulli NB with Laplace smoothing; absent-feature terms matter.
features=np.array([[0,0],[0,1],[1,0],[1,1]])
classes=np.array([0,0,1,1])
prob=np.stack([(features[classes==j].sum(0)+1)/(sum(classes==j)+2) for j in (0,1)])
log_scores=np.stack([(features*np.log(p)+(1-features)*np.log1p(-p)).sum(1)+np.log(.5)
                     for p in prob],axis=1)
assert np.array_equal(log_scores.argmax(1),classes)
print('linear, calculus, Bayes, logistic, GDA and NB fixtures pass')

For logistic loss, gradient is X.T@(sigmoid(Xtheta)-y)/m and Hessian X.T diag(p*(1-p)) X/m. Separation can make unregularized MLE coefficients diverge even when classification improves; use regularization and held-out calibration. Locally weighted regression refits around each query with a bandwidth; it is not one global theta and can become ill-conditioned with tiny effective sample support. GDA models p(x|y)p(y) with Gaussian class-conditionals, while logistic directly models p(y|x). Naive Bayes' conditional-independence assumption is often false yet can produce a useful baseline; choose Bernoulli versus count/multinomial semantics explicitly.

Course roadmap: derivations and implementation contracts

The following table preserves the original study goals without inventing completed posts. Use topic names rather than assuming one edition's lecture numbers apply everywhere. For each topic, produce a short summary, a derivation, a NumPy implementation, a generated or authorized real-data example and a factual “where I got stuck” note. Mark results only after execution; do not reconstruct a personal story from a template.

An exponential-family model has density/mass b(y) exp(eta*T(y)-a(eta)). A canonical GLM sets eta=theta.T x and links prediction to the mean determined by a's derivative under regularity. Gaussian and Bernoulli examples recover familiar regression rules; the link is not interchangeable across response distributions.

A perceptron updates on a mistaken signed example: w += eta*y*x for y in {−1,+1}. Convergence requires linear separability and other standard assumptions; a nonseparable dataset can cycle. An SVM instead minimizes a regularized hinge objective or solves its constrained margin problem. Its kernel dual uses K_ij=phi(x_i).T phi(x_j); an arbitrary similarity need not be a PSD kernel. SMO optimizes two dual coordinates because one alone cannot preserve sum alpha_i*y_i=0: choose a pair, derive feasible bounds, optimize the one-dimensional quadratic, clip, update the paired coordinate and bias, then check KKT residuals. This is a solver study task, not a false claim that the primal example below implements SMO.

Further runnable algorithms and independent small oracles

# Primal regularized hinge classifier; not a kernel/SMO implementation.
features=np.array([[-2.,-1.],[-1.,-2.],[1.,2.],[2.,1.]])
signs=np.array([-1.,-1.,1.,1.])
w=np.zeros(2)
for t in range(200):
    violating=signs*(features@w)<1
    gradient=.1*w-(signs[violating,None]*features[violating]).sum(0)/len(features)
    w-=.1*gradient
assert np.array_equal(np.sign(features@w),signs)
# Two-layer tanh network, scalar squared loss and all-coordinate derivatives.
rng=np.random.default_rng(229)
A=rng.normal(size=(2,3))*.1;B=rng.normal(size=(3,1))*.1
x=features[:2];target=np.array([[.2],[-.1]])
def network_loss(A,B): return .5*np.mean((np.tanh(x@A)@B-target)**2)
h=np.tanh(x@A);error=(h@B-target)/len(x)
gB=h.T@error;gA=x.T@((error@B.T)*(1-h*h))
for weights,analytic in [(A,gA),(B,gB)]:
    for ix in np.ndindex(weights.shape):
        old=weights[ix];weights[ix]=old+eps;plus=network_loss(A,B)
        weights[ix]=old-eps;minus=network_loss(A,B);weights[ix]=old
        assert abs((plus-minus)/(2*eps)-analytic[ix])<1e-8
# K-means: deterministic initial centroids and explicit empty-cluster retention.
points=np.array([[-2.,0.],[-1.,0.],[1.,0.],[2.,0.]])
centers=points[[0,3]].copy();objectives=[]
for _ in range(5):
    distance=((points[:,None]-centers[None,:])**2).sum(2)
    assignment=distance.argmin(1)
    objectives.append(distance.min(1).sum())
    centers=np.stack([points[assignment==j].mean(0) if np.any(assignment==j) else centers[j] for j in range(2)])
assert all(b<=a+1e-12 for a,b in zip(objectives,objectives[1:]))
np.testing.assert_allclose(centers,[[-1.5,0],[1.5,0]])
# Two-component 1D Gaussian EM with a variance floor; generated fixture only.
v=np.array([-2.2,-2.,-1.8,1.8,2.,2.2]);mu=np.array([-1.,1.]);variance=np.ones(2);mix=np.ones(2)/2
likelihood=[]
for _ in range(15):
    logp=np.log(mix)[None,:]-.5*(np.log(2*np.pi*variance)[None,:]+(v[:,None]-mu)**2/variance)
    maximum=logp.max(1,keepdims=True)
    logsum=maximum+np.log(np.exp(logp-maximum).sum(1,keepdims=True))
    responsibility=np.exp(logp-logsum)
    likelihood.append(logsum.sum())
    count=responsibility.sum(0)
    mu=(responsibility*v[:,None]).sum(0)/count
    variance=np.maximum((responsibility*(v[:,None]-mu)**2).sum(0)/count,1e-6)
    mix=count/len(v)
assert all(b>=a-1e-10 for a,b in zip(likelihood,likelihood[1:]))
np.testing.assert_allclose(np.sort(mu),[-2,2],atol=1e-6)
# PCA requires centering; sign of a component is arbitrary.
centered=points-points.mean(0)
_,singular,vt=np.linalg.svd(centered,full_matrices=False)
projection=centered@vt[:1].T
np.testing.assert_allclose(projection@vt[:1],centered,atol=1e-12)
# Two-state discounted MDP: action0 waits, action1 enters rewarding absorbing state.
gamma=.9
transition=np.array([[[1,0],[0,1]],[[0,1],[0,1]]],dtype=float)
reward=np.array([[0.,1.],[0.,0.]])
value=np.zeros(2)
for _ in range(100): value=(reward+gamma*np.einsum('sak,k->sa',transition,value)).max(1)
np.testing.assert_allclose(value,[1.,0.])
policy=(reward+gamma*np.einsum('sak,k->sa',transition,value)).argmax(1)
P=transition[np.arange(2),policy];r=reward[np.arange(2),policy]
np.testing.assert_allclose(np.linalg.solve(np.eye(2)-gamma*P,r),value)
# Terminal Q-learning target excludes bootstrapping.
q=np.zeros((2,2));q[1]=100
s,a,r,next_s,terminal=0,1,1.,1,True
q[s,a]+=.5*(r+(0 if terminal else gamma*q[next_s].max())-q[s,a])
assert q[0,1]==.5
print('hinge, network gradients, kmeans, EM, PCA, Bellman and terminal TD checks pass')

The MDP fixture pays reward 1 on the transition from state 0 via action 1; the absorbing state itself pays 0 thereafter. The “rewarding absorbing state” comment is shorthand for entry reward, not reward 1 at every future step (which would give a different value). Its transition rows sum to 1 and its Bellman value [1, 0] follows that explicit reward table.

K-means optimizes squared Euclidean distortion by local alternating minimization, not a guaranteed global clustering. EM increases its likelihood under exact valid updates but can reach local optima; Gaussian-mixture likelihood can become singular without constraints. A variance floor changes the optimization contract and should be reported. PCA maximizes projected variance after centering, not necessarily class separability; eigenvector signs are arbitrary. ICA seeks independent latent sources with stronger assumptions such as non-Gaussianity; uncorrelated PCA coordinates are not generally independent. Factor analysis models x=mu+Wz+epsilon with latent Gaussian z and typically diagonal noise covariance; it distinguishes shared latent variation from measurement noise.

Trees choose splits by a criterion such as weighted impurity or squared error. Test every candidate threshold on a tiny dataset as an oracle; guard empty leaves and train-only preprocessing. Bagging reduces some estimator variance by bootstrap aggregation; boosting sequentially fits residual/gradient structure and can overfit. Neither guarantees improved test error. A model-comparison table should include assumptions, objective, prediction cost, interpretability and failure modes, not a universal ranking.

Learning theory, model selection and practical debugging

Bias/variance decomposition for squared error uses a distribution over training sets and a specified data-generating process. High training/validation error can suggest underfitting or optimization/data defects; a gap can suggest overfitting or distribution mismatch. Curves are diagnostic hypotheses, not unique diagnoses. VC dimension measures shattering capacity of a hypothesis class, not the number of parameters in every model. PAC-style generalization bounds require explicit sampling/class/loss assumptions and may be loose for a particular implementation.

Split by subject/document/time when those induce dependence. Fit normalization, feature selection and imputation on training folds only. Tune hyperparameters on validation or inner CV; use outer folds to assess the selection procedure without reusing its inner tuning scores, while reporting finite-sample and distribution limitations. Leave a final test set untouched. For temporal deployment, random CV may leak future information. Report uncertainty and subgroup behavior, not just the best seed.

Start debugging with label/feature alignment, a constant baseline, a tiny overfit example, finite gradients and an independently calculated metric. Then examine failure categories and learning curves. Ceiling analysis asks how much a perfect component could improve the whole system; replacing one component with an oracle is an experiment, not a promise that its real replacement reaches that ceiling.

For an MDP with known transitions, Bellman's optimality operator contracts in the sup norm for discount 0 ≤ gamma < 1, making value iteration well-defined in the standard finite bounded setting. Policy iteration alternates evaluation (solve the linear system) and improvement. Q-learning estimates action values from sampled transitions; exploration and learning-rate conditions matter, especially off-policy. Policy gradients use grad log pi(a|s) weighted by return/advantage; a state-dependent baseline can reduce variance without bias under the usual action-independence condition. LQR/LQG and POMDPs add dynamics/partial-observation assumptions and are later study branches, not equivalent to this two-state fixture.

Python practice, worked exercises and study progression

Use array shape assertions and NumPy broadcasting deliberately. y of shape(m,) minus prediction(m,1) creates an m×m matrix; reshape consistently. Prefer solve/lstsq to explicit inverses, eigh for real symmetric matrices and svd for rectangular rank analysis. Small loops are fine for gradient checks and reference oracles; “avoid all Python loops” is not a correctness rule.

Matplotlib can visualize loss, decision boundaries and singular-value spectra, but label whether a plot is generated/synthetic or measured. A notebook is an exploration interface, not a reproducibility guarantee: restart the kernel and run top-to-bottom, then preserve an executable script, environment and input hashes. The examples above are numerical fixtures; plotting and notebook setup are optional follow-on work.

1. Why can least squares have many optima? If X has a nonzero null vector v, theta and theta+v have the same predictions. Convexity does not imply uniqueness. The duplicated-column fixture demonstrates it.
2. When does a Hessian establish a minimum? At a stationary point, PD curvature gives a strict local minimum under smoothness. PSD everywhere on a convex domain establishes convexity, not automatic convergence for arbitrary step sizes.
3. Why is MAP like ridge here? Gaussian noise yields squared residuals; a Gaussian coefficient prior yields a quadratic penalty. Its coefficient depends on variances and loss normalization, and the intercept policy must be declared.
4. GDA versus logistic? One models class-conditional features and uses Bayes; the other directly models class probability. Misspecified Gaussian assumptions and limited data change their relative strengths.
5. Why not call the roadmap a complete course summary? It names the learning route and supplies representative implementations/derivations, while separate lecture-level projects remain study tasks. This orientation is a starting point, not a substitute for those later projects.
6. What should a future study entry contain? One-page summary, re-derived central equation, runnable implementation with independent checks, a generated or authorized dataset example and an honest friction-point note. Add a comparison table and link only to work that actually exists.

Suggested progression: first replay the prerequisite lab; next derive logistic/Newton/GDA and margin methods; then complete a supervised project with leakage-safe selection; then compare K-means/EM/PCA on an authorized dataset; finally implement a small control problem with a known Bellman oracle. Keep the original planned topics—local regression, GLMs, SVM/SMO, backprop, bias/variance, EM and RL—as concrete study tasks rather than fabricated historical achievements.

Sources and verification boundary

Official2018 syllabus, CS229 consolidated notes,23August2026 snapshot, official linear algebra review. The CPU fixtures validate the displayed generated examples; they are not a Stanford credential, a complete implementation of every advanced algorithm, or a real-dataset benchmark.

Review scope on 10 October 2026: the existing programs remain byte-for-byte intact. This pass checks source, shapes, fixed-function derivatives and bounded arithmetic only; it does not run fitting/training loops, download models/data, or award course completion. Roadmap items, comparison tables and future lecture posts remain study tasks; publication/selector decisions belong to the site owner.