Kimi K3 From Zero to Deep — Math, Architecture, Training, and Systems

Sources (please open these too):

• Moonshot AI blog: kimi.com/blog/kimi-k3
• Technical report (PDF): github.com/MoonshotAI/Kimi-K3
• Model weights: huggingface.co/moonshotai/Kimi-K3
• Related open-source projects referenced: FlashKDA, MiniTriton, nano-KPU, AgentENV
• All figures/images referenced in this post live in the original blog page — open the Moonshot blog in a second tab to see them next to each explanation.

One-line summary. Kimi K3 is a 2.8-trillion-parameter Mixture-of-Experts language model with native vision and a 1-million-token context window, released open-weights by Moonshot AI. It is the first open model in the "3-trillion-parameter class," and its main technical contributions are a new attention mechanism (Kimi Delta Attention), a new way of routing information across layers (Attention Residuals), and a stabilized extreme-sparsity Mixture-of-Experts (Stable LatentMoE) that together give it ~2.5× the scaling efficiency of Kimi K2.

That sentence is dense. The rest of this post unpacks every term in it — starting from "what is attention" — so that by the end you can read the actual tech report and understand what each equation is doing and why.

---

0. How to read this post

This post is written as a layered explainer. Each big idea is introduced three times:

1. Story mode — plain English, no math, why we care.
2. Diagram mode — a picture with labeled boxes.
3. Technical mode — the actual equation or algorithm, with every symbol defined.

If a section gets hairy, you can safely skip the "Technical mode" boxes and still follow the story. If you only care about the math, jump to the technical boxes.

L3

Story mode

one-paragraph plain English

L2

Diagram mode

boxes and arrows

L1

Technical mode

equations with every symbol defined

core

---

1. The one-page TL;DR

L9

2.8T total parameters, 104B activated per token

Mixture-of-Experts sparsity ≈ 27×

L8

1,000,000-token context window

long-horizon coding + agentic work

L7

Native vision (MoonViT-V2 encoder)

images + video in the same model

L6

Kimi Delta Attention + Gated MLA (hybrid)

linear-time long-range mixing + occasional full attention

L5

Attention Residuals (AttnRes)

every layer can read every earlier block

L4

Stable LatentMoE: 16 of 896 routed experts

SiTU-GLU + RMSNorm + Quantile Balancing

core

L3

MXFP4 weights / MXFP8 activations from SFT onward

quantization-aware training for cheap serving

L2

RL across coding, agent, knowledge domains

unified white-box RL environment + on-policy distillation

L1

~2.5× scaling efficiency vs. Kimi K2

more intelligence per unit of compute

The reported headline results (from Figure 1 of the tech report): Kimi K3 sits behind Claude Fable 5 and GPT-5.6 Sol on most benchmarks, but is ahead of every other open and proprietary model tested — including GLM-5.2, GPT-5.5, and Claude Opus 4.8. On some benchmarks (BrowseComp, AutomationBench, ProgramBench, SWE-Marathon), it is the top-scoring model.

Now let’s actually understand why those numbers came out that way.

---

2. Background — what is a language model even doing?

Skip this section if you already know the basics of transformers.

2.1 Language modeling in one sentence

A language model is a function that reads a sequence of tokens (roughly, "word pieces") and predicts the probability distribution of the next token. Everything else — chatting, coding, "reasoning" — is downstream of this single objective.

📥

Tokens in

"The cat sat on the"

Embeddings

vectors of numbers

N transformer blocks

attention + FFN, repeated

Logits

score per vocab token

📤

Next token

"mat"

1. 1
   Tokens in

   "The cat sat on the"
2. 2
   Embeddings

   vectors of numbers
3. 3
   N transformer blocks

   attention + FFN, repeated
4. 4
   Logits

   score per vocab token
5. 5
   Next token

   "mat"

Everything hard about scaling a language model is (a) how the "N transformer blocks" mix information along the sequence, and (b) how much useful compute you can pack into each block.

2.2 The three axes of scale

The Kimi team frames K3 as scaling three complementary dimensions:

L3

Sequence length

how far back the model can look — K3 pushes to 1M tokens via KDA + Gated MLA

L2

Depth

how many layers of processing — AttnRes lets every layer read every earlier block

L1

Width

how many parallel specialists — Stable LatentMoE grows the expert pool to 896

core

Keep this triangle in mind. Every architectural choice below is aimed at one of these three axes.

2.3 Standard attention, in one paragraph

Given a sequence of token vectors, self-attention lets each token pull information from every other token by (i) projecting each vector into three roles — Query (Q), Key (K), Value (V), (ii) computing a similarity score between each Query and every Key, (iii) softmax-normalizing those scores, and (iv) using them as weights to average the Values.

$$\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right) V$$

The problem: computing $QK^\top$ for a sequence of length $L$ costs $\mathcal{O}(L^2)$ time and memory. At $L = 1{,}000{,}000$ this is a trillion pairwise comparisons per layer per head. You cannot brute-force your way to million-token context. This is why Kimi K3 needs a smarter attention.

2.4 Tokens, vocabulary, and embeddings

A model does not read characters or words directly. A tokenizer converts text into integer IDs from a fixed vocabulary. A token may be a whole word, part of a word, punctuation, whitespace, or a byte sequence. K3 uses a vocabulary of roughly 160,000 entries. The same sentence can therefore occupy different numbers of tokens under different tokenizers.

If token ID $i$ enters the model, an embedding table $E \in \mathbb{R}^{|V| \times d}$ selects row $E_i \in \mathbb{R}^{d}$. That row is not a definition of the token. It is a learned coordinate: a point in a high-dimensional space whose geometry becomes useful during training.

Three terms that are easy to confuse:

• Token — one discrete vocabulary item.
• Embedding — the dense vector representing that token at the model entrance.
• Hidden state — the evolving vector for that token after one or more layers have mixed information into it.

A hidden width of $d=7168$ means every textual position is represented by 7,168 floating-point coordinates inside K3. The model repeatedly transforms those coordinates; it does not repeatedly manipulate the original text.

Watch first: Let’s build the GPT Tokenizer — Andrej Karpathy. The tokenizer is outside the neural network, but every context-length, cost, and latency number is measured in its output tokens.

2.5 The minimum linear algebra

You only need four objects to read most of the K3 equations:

A matrix multiplication $y=Wx$ is a learned change of coordinates. Each output coordinate is a weighted sum of all input coordinates:

$$y_i = \sum_{j=1}^{d} W_{ij}x_j.$$

The dot product $q^\top k=\sum_j q_jk_j$ measures directional alignment. Large positive alignment means the vectors point similarly; a negative value means opposing directions. Attention turns these alignments into retrieval weights.

The outer product $kv^\top$ is different. If $k \in \mathbb{R}^{d_k}$ and $v \in \mathbb{R}^{d_v}$, then $kv^\top \in \mathbb{R}^{d_k \times d_v}$. It creates a matrix that associates a key direction with a value. KDA writes exactly this object into its recurrent memory.

The identity matrix $I$ leaves a vector unchanged. Therefore $I-\beta kk^\top$ means “keep everything except some component along $k$.” That is why it acts as KDA’s erase operator.

A diagonal matrix $\operatorname{Diag}(\alpha)$ scales each coordinate independently. KDA’s channel-wise forget gate uses it to retain some memory channels longer than others.

Watch visually:

1. But what is a neural network? — 3Blue1Brown
2. Gradient descent, how neural networks learn — 3Blue1Brown
3. Backpropagation calculus — 3Blue1Brown

2.6 Probability, logits, softmax, and temperature

The model ends each step with one unnormalized score, called a logit, for every vocabulary token. If $z_i$ is token $i$’s logit, softmax converts all logits into a probability distribution:

$$p_i = \frac{\exp(z_i/T)}{\sum_j \exp(z_j/T)}.$$

• $p_i \ge 0$ and $\sum_i p_i=1$.
• $T$ is temperature. Smaller $T$ sharpens the distribution; larger $T$ flattens it.
• Adding the same constant to every logit changes no probability. Implementations exploit this by subtracting $\max_j z_j$ before exponentiating, preventing overflow.

The same softmax idea appears twice in this post:

1. Vocabulary softmax chooses the next token.
2. Attention softmax chooses how much information to retrieve from each position or depth block.

A categorical distribution is simply a probability over discrete choices. Sampling from it creates variation; choosing $\arg\max_i p_i$ is deterministic greedy decoding.

2.7 Loss, information, entropy, and perplexity

For a training example whose correct next token is $y$, the negative log-likelihood or cross-entropy loss is

$$\mathcal{L}_{\text{NLL}} = -\log p(y \mid x_{\le t}).$$

This is measured in nats when the logarithm is natural. It assigns a small penalty to a confident correct prediction and an enormous penalty to a confident wrong prediction. Averaging it over tokens gives the language-model training loss.

Entropy measures uncertainty inside a distribution:

$$H(p) = -\sum_i p_i \log p_i.$$

A distribution concentrated on one token has low entropy; a flat distribution has high entropy. Perplexity is $\exp(\text{average loss})$. It can be read loosely as the model’s effective number of plausible next-token choices, but only when comparing the same tokenizer and data distribution.

KL divergence measures how differently two distributions assign probability:

$$D_{\mathrm{KL}}(p\|q)=\sum_i p_i\log\frac{p_i}{q_i}.$$

K3’s multi-teacher distillation uses a clipped token-level log-ratio related to this quantity. The ratio tells the student whether its probability for the sampled token is below or above the appropriate expert teacher’s probability.

2.8 Gradients, backpropagation, and optimizers

A gradient $\nabla_\theta \mathcal{L}$ is the vector of partial derivatives of the loss with respect to every parameter $\theta$. It points toward the direction of fastest local increase, so gradient descent moves the other way:

$$\theta_{t+1}=\theta_t-\eta\,\nabla_\theta\mathcal{L},$$

where $\eta$ is the learning rate. Backpropagation applies the chain rule from the loss backward through every operation to compute those derivatives efficiently.

Real optimizers add memory and normalization:

• Momentum averages recent gradients, reducing noisy zig-zag motion.
• Adam/AdamW track per-parameter first and second moments and scale coordinates independently.
• Muon orthogonalizes matrix-valued momentum updates so singular directions receive a better-balanced update.
• Per-Head Muon, K3’s refinement, performs that orthogonalization separately for each attention head instead of coupling all heads in one matrix.

The scientific idea is conditioning. If a loss surface is extremely steep in one direction and flat in another, one global step size performs badly. Normalization and optimizer geometry make progress less sensitive to that mismatch.

Code-level foundation: The spelled-out intro to neural networks and backpropagation: building micrograd — Andrej Karpathy.

2.9 Neural-network anatomy: attention, FFN, residual, normalization

A Transformer layer alternates two kinds of computation:

1. Attention mixes information across token positions. It answers “which earlier positions should this token read?”
2. The feed-forward network (FFN) mixes information across channels within each token. It answers “what nonlinear features should this token compute after reading?”

Each sublayer is wrapped by a residual connection. Instead of replacing $x$ with $f(x)$, it forms roughly $x+f(x)$. This creates a direct path for information and gradients through depth.

Normalization controls scale. RMSNorm divides by the root mean square of the coordinates:

$$\operatorname{RMSNorm}(x)=g\odot\frac{x}{\sqrt{\frac{1}{d}\sum_{j=1}^{d}x_j^2+\epsilon}},$$

where $g$ is learned and $\epsilon$ prevents division by zero. Unlike LayerNorm, RMSNorm does not subtract the mean.

A head is one independent attention subspace. Multiple heads let the layer represent several retrieval patterns at once. “96 attention heads” does not mean 96 separate models; they are parallel slices inside one layer.

Best conceptual pair:

• Transformers, the tech behind LLMs — 3Blue1Brown
• Attention in transformers, step-by-step — 3Blue1Brown

2.10 Dense models, sparse models, and Mixture-of-Experts

A dense model runs every parameter in every layer for every token. A Mixture-of-Experts layer keeps many alternative FFNs called experts and uses a router to select only a few for each token.

If a layer has $N$ routed experts and chooses top-$k$, then its expert activation fraction is $k/N$. K3 chooses 16 of 896 routed experts, so only about 1.8% of routed experts execute for one token. Shared experts still run for every token.

This explains why three parameter counts can all be valid:

• Total parameters describe storage and potential capacity.
• Activated parameters describe roughly how much of the model participates per token.
• FLOPs per token describe arithmetic, which also depends on sequence length, attention, batch shape, and implementation.

MoE is not free. The router can overload popular experts; tokens and activations must move between GPUs; underused experts learn slowly. K3’s LatentMoE, Quantile Balancing, and MoonEP address those separate problems.

Watch: What are Mixture-of-Experts Models — Hugging Face.

2.11 Sequence complexity, memory, bandwidth, and precision

Big-O notation describes how cost grows, not the exact runtime. For sequence length $L$:

• Full attention materializes an $L\times L$ score pattern, so its pairwise work is $\mathcal{O}(L^2)$.
• Recurrent or linear attention updates fixed-size state once per token, so sequence scaling is $\mathcal{O}(L)$.
• A KV cache grows with generated context because full-attention layers retain keys and values per token.
• KDA instead carries a fixed-size recurrent state per layer, though K3 still has periodic MLA layers with a compressed KV cache.

On modern accelerators, memory bandwidth often matters as much as arithmetic. Moving a weight from HBM can cost more time than multiplying it. MoE reduces arithmetic per token but introduces all-to-all communication. Quantization stores more weights per byte and moves less data.

Floating-point formats trade range and precision:

• FP32: wide and accurate, expensive.
• BF16/FP16: common training compute formats.
• FP8: fewer bits, higher throughput, tighter numerical constraints.
• MXFP4: block-scaled 4-bit format used for K3’s routed expert weights during deployment-aware post-training.

A block scale lets a small group of low-bit numbers share a higher-precision scale. Quantization-aware training simulates those rounding and saturation effects while learning, giving the model a chance to adapt instead of compressing it only after training.

2.12 Reinforcement learning vocabulary before K3’s post-training section

K3’s post-training report assumes a compact RL vocabulary:

• Policy $\pi_\theta$ — the model’s probability distribution over next actions or tokens.
• Trajectory / rollout — one complete interaction: prompts, reasoning, tool calls, observations, and answer.
• Reward — a scalar signal saying how desirable an outcome or action was.
• Verifier — program or judge that decides whether an output satisfies a criterion.
• On-policy — data sampled from the policy currently being optimized.
• Off-policy / stale data — data produced by an older policy.
• Credit assignment — deciding which earlier actions caused a later success or failure.
• Distillation — training a student to match one or more teacher distributions.
• Reward hacking — exploiting the reward rule without solving the intended task.

K3’s agent trajectories can span many tool calls and training iterations. Partial rollouts reduce waiting for stragglers, but create stale data. Token-level regularization constrains how far the policy moves, while budget penalties discourage unnecessary reasoning or verbosity.

Watch: Reinforcement Learning with Human Feedback, Clearly Explained — StatQuest. K3’s exact algorithm is broader than classic RLHF, but this establishes policy, reward model, preference comparison, and optimization.

2.13 A worked shape trace through one attention head

Suppose a toy batch has $B=2$ sequences, each $L=4$ tokens, hidden width $d=8$, and $H=2$ heads. Each head has $d_k=d_v=4$.

1. Input hidden states: $X \in \mathbb{R}^{2\times4\times8}$.
2. Linear projections: $Q,K,V \in \mathbb{R}^{2\times2\times4\times4}$ after splitting into heads.
3. Full-attention scores: $QK^\top \in \mathbb{R}^{2\times2\times4\times4}$.
4. Softmax runs across the final key-position axis.
5. Weighted values return $O \in \mathbb{R}^{2\times2\times4\times4}$.
6. Concatenate heads to recover $\mathbb{R}^{2\times4\times8}$.

At one million tokens, the score matrix for one sequence and one head would conceptually have $10^{12}$ entries. At two bytes each, merely storing that matrix would require about 2 TB before accounting for other heads, layers, gradients, or workspace. FlashAttention avoids materializing the full matrix, but it cannot remove the underlying quadratic pairwise computation. KDA changes the algorithm instead.

2.14 What “science” means in this report

K3 combines four kinds of scientific evidence:

1. Mathematical derivation — equations specify recurrences, losses, and communication operators.
2. Controlled ablation — change one component at smaller scale and compare validation loss or downstream metrics.
3. Scaling-law experiment — fit behavior across model size, token budget, batch size, and learning rate, then choose a large-run recipe.
4. Systems measurement — benchmark kernels, communication, cache hit behavior, sandbox latency, and end-to-end throughput.

None alone proves general intelligence. Benchmarks are empirical measurements under a particular harness, prompt, reasoning budget, tool set, and date. A score difference can come from the base model, the agent harness, context management, tools, or the evaluator. Read K3’s benchmark table as evidence about a complete evaluated system—not a timeless ranking of isolated neural networks.

---

3. Architecture overview

Kimi K3's block is unusual. Each block contains three KDA layers followed by one Gated MLA layer, and each attention layer is paired with a Stable LatentMoE feed-forward network. Above that, Attention Residuals rewire how blocks talk to each other, and at the bottom MoonViT-V2 pipes vision into the same embedding space.

Read that diagram carefully — it is the whole model in one picture. The next four sections zoom into each colored region.

3.1 Exact model shape — keep these counts separate

These numbers describe different axes. The 93 sequential layers are partitioned into eight depth groups for AttnRes. Within the layer stack, KDA is the dominant token-mixing mechanism and MLA appears periodically as a full-attention checkpoint. Do not multiply “eight groups” by “four layer types” and conclude the model has 32 layers—the diagram is conceptual, not a literal layer count.

The routed expert fraction is $16/896 \approx 1.79\%$ per token. That does not mean only 1.79% of the entire network runs: shared experts, attention, routers, embeddings, and other dense components also execute. The report’s 52B activated count is the useful end-to-end number.

---

4. Kimi Delta Attention (KDA) — linear-time attention with a "forget gate"

Story mode. Standard attention is $\mathcal{O}(L^2)$. Linear attention is a family of tricks that reduces this to $\mathcal{O}(L)$ by rewriting attention as a recurrence: instead of comparing every token to every other token, you maintain a compact "state matrix" that summarizes everything you have seen so far, and you update it one token at a time. KDA is Moonshot's tuned version of one of these tricks, called the delta rule, plus a channel-wise forget gate.

If that felt fast, don't worry — we'll unpack every word.

4.1 Why linear attention exists at all

The classical result behind linear attention is that if you replace the $\text{softmax}$ in attention with a kernel $\phi$ that factorizes, you can rewrite

$$o_t = \sum_{i \le t} \phi(q_t)^\top \phi(k_i)\, v_i^\top = \phi(q_t)^\top S_t, \qquad S_t = S_{t-1} + \phi(k_t) v_t^\top.$$

Read this slowly. The right-hand side says: to compute the output at token $t$, you don't need every previous key and value — you only need $S_t$, a $d_k \times d_v$ matrix that has been rolled forward one token at a time. Each new token adds $\phi(k_t)v_t^\top$ to $S_t$. That is a recurrent neural network, not a transformer.

Cost: $\mathcal{O}(L \cdot d_k d_v)$ — linear in sequence length. Memory: constant per layer.

Problem with vanilla linear attention: because $S_t$ just accumulates forever, the model has no way to "forget" old information. Long sequences drown in stale context.

4.2 The delta rule — writing over yourself instead of adding

The delta rule (Schlag et al., 2021) fixes this by making the recurrence look like an error-correcting write:

$$S_t = \left(I - \beta_t k_t k_t^\top\right) S_{t-1} + \beta_t k_t v_t^\top.$$

The scalar $\beta_t \in (0, 1)$ is a per-token "write strength." Read this as: "erase whatever the current key would have retrieved from the old state, then write in the new value." It behaves much more like a mutable dictionary and less like an infinite accumulator.

4.3 KDA's contribution — channel-wise forgetting

KDA (Kimi Team, Kimi Linear, 2025) adds one more piece: a channel-wise decay vector $\alpha_t \in (0, 1)^{d_k}$. Each channel (each row of the state) is multiplied by its own retention factor before the delta rule fires. This lets the model forget some dimensions of the state faster than others.

$$\boxed{\; S_t \;=\; \left(I - \beta_t k_t k_t^\top\right)\,\text{Diag}(\alpha_t)\, S_{t-1} \;+\; \beta_t k_t v_t^\top, \quad \tilde o_t = S_t^\top q_t. \;}$$

The Kimi K3 report calls this Equation (1). Every symbol:

• $x_t \in \mathbb{R}^d$ — the hidden state of token $t$.
• $q_t, k_t \in \mathbb{R}^{d_k}$ — query and key, both derived from $x_t$.
• $v_t \in \mathbb{R}^{d_v}$ — value, also from $x_t$.
• $S_t \in \mathbb{R}^{d_k \times d_v}$ — the recurrent state (small, fixed-size).
• $\alpha_t$ — per-channel retention (learned, per-token, per-head).
• $\beta_t$ — per-token write strength.
• $I - \beta_t k_t k_t^\top$ — an "erase" projection along the current key.

The projections use ShortConv → Swish → L2Norm, which are just a small 1D convolution, a smooth nonlinearity, and vector-length normalization. These make the queries and keys more stable numerically.

4.4 Why Kimi K3 changed the log-decay parameterization

The tricky part in software is computing $S_t$ in parallel over a chunk of tokens, not one at a time. To do that, you need cumulative products of the decays, and those products can become astronomically small or large — overflowing in fp16/bf16.

Kimi Linear (K2) mapped the raw decay logit $z$ through a negative-softplus: $g = -e^A \, \text{softplus}(z)$. This can go to $-\infty$, so the reciprocal $1/\gamma$ can overflow. Their fix was expensive: split each chunk into 16-token tiles and do the diagonal tiles with explicit position-pair loops that cannot use tensor cores.

Kimi K3 replaces this with a scaled sigmoid that bounds the log-decay from below:

$$g_t^h \;=\; g_{\min}\, \text{Sigmoid}(e^{A_h} z_t^h) \in (g_{\min}, 0)^{d_k}, \qquad g_{\min} = -5.$$

The payoff: the whole KDA kernel now runs as dense matmul on Tensor Cores. That is why K3 gets ~2.5× the effective throughput of K2.

4.5 Chunkwise parallel form (skip if math-averse)

To train efficiently you need to process a chunk of $C$ tokens in parallel. The trick is to compute the intra-chunk contribution as a masked matmul, and the inter-chunk contribution as a state carried between chunks. The tech report’s Equation (4) is:

$$O_{[t]} \;=\; \underbrace{(\Gamma^{1\to C}_{[t]} \odot Q_{[t]})\, S_{[t]}}_{\text{inter-chunk}} \;+\; \underbrace{A_{[t]}\, \widetilde V_{[t]}}_{\text{intra-chunk}}, \quad A_{[t]} = \text{Tril}\!\left[(Q_{[t]}\!\odot\!\Gamma^{1\to C}_{[t]})(K_{[t]}/\Gamma^{1\to C}_{[t]})^\top\right].$$

The Tril mask enforces causality (a token can only attend to itself and earlier tokens). The bounded decay guarantees the $1/\Gamma$ division does not blow up.

4.6 Gated MLA — the "occasional global attention"

KDA is fast but its state is a fixed-size matrix, which limits precision on very hard lookup tasks. So K3 interleaves one Gated MLA layer for every three KDA layers. MLA = Multi-head Latent Attention (from DeepSeek-V2), which is standard $\mathcal{O}(L^2)$ attention but with a compressed key/value cache. "Gated" adds a learned gate so the attention output can be attenuated when it isn't useful.

The intuition: KDA is a fast summarizer; Gated MLA is a precise retriever. Three summarizers per retriever is a compute-quality sweet spot Moonshot found empirically.

---

5. Attention Residuals (AttnRes) — every layer talks to every earlier block

Story mode. In a normal transformer, information flows straight up the stack via residual connections: layer $\ell$'s input is layer $\ell{-}1$'s output. That means information from layer $3$ has to be re-encoded through layers $4, 5, \ldots$ if a later layer wants to use it. Attention Residuals let a later layer directly attend to the outputs of any earlier block — like a shortcut network across depth.

5.1 Standard residuals vs. attention residuals

5.2 The AttnRes equations

Each layer stores a learned pseudo-query $w$. That query attends over the block outputs $[b_0, b_1, \ldots, b_{n-1}]$ (where $b_0$ is the embedding), producing attention weights $\alpha$. The residual added to the current layer's output is a weighted mix of those past block outputs.

$$\alpha \;=\; \text{softmax}(w^\top B),\qquad \text{residual} \;=\; B\, \alpha, \qquad B = [b_0, b_1, \ldots, b_{n-1}].$$

Crucially:

• Attention weights $\alpha$ are shared per position and there is one $w$ per layer, so this adds negligible parameters.
• Attention is over blocks (chunks of layers), not individual layers, keeping memory at $\mathcal{O}(Nd)$ where $N \approx 8$ is the number of blocks.
• Empirically $N \approx 8$ recovers most of the benefit; K3 uses 8 blocks of 12 layers each ≈ 96 layers total.

Why it matters. Deep networks tend to "forget" early features because gradient signals attenuate. AttnRes gives each layer explicit read access to earlier abstractions — a bit like a residual highway plus selective routing. In the tech report, ablations show AttnRes contributes a meaningful chunk of the 2.5× scaling-efficiency win.

---

6. Stable LatentMoE — 16 experts out of 896

Story mode. A "Mixture of Experts" replaces a single big feed-forward network in each block with many small ones ("experts"), plus a router that picks which few to run for each token. The result: total parameters go up (more knowledge storage), but compute per token stays flat (only a handful of experts activate). Kimi K3 pushes this to an extreme — 896 routed experts, 16 activated per token, sparsity ratio 56. That is roughly 3–4× more expert diversity than any published open model.

Extreme sparsity is hard for two reasons:

1. Activation explosion. Chained matmuls in the routed branch amplify outliers.
2. Load balancing. With 896 experts, the router will inevitably neglect some and overload others.

Stable LatentMoE fixes both. Let's take them one at a time.

6.1 LatentMoE — cheap experts via a shared bottleneck

Vanilla MoE hands each expert the full $d$-dimensional hidden state, and each expert has its own $d \to 4d \to d$ MLP. If you activate 16 experts, you pay 16× the communication and expert weight traffic per token.

LatentMoE (Elango et al., 2026) splits the model width into two pieces:

• A shared full-width path (2 shared experts in K3) that always runs.
• A compact latent path at width $\ell \ll d$: a shared projection $W^\downarrow$ takes the token down to $\ell$; each routed expert operates in $\mathbb{R}^\ell$; a single $W^\uparrow$ maps the aggregated result back to $\mathbb{R}^d$.

$$u \;=\; \sum_{i \in \mathcal T_k(x)} p_i\, E_i^{\text{routed}}(W^\downarrow x), \qquad y \;=\; \sum_{j=1}^{N_s} E_j^{\text{shared}}(x) \;+\; W^\uparrow\, \text{RMSNorm}(u).$$

This is Equation (11) in the tech report. It is the payoff of the whole design: you can now afford 896 routed experts because they operate in a narrow latent space.

6.2 Normalized LatentMoE — one extra RMSNorm

The routed aggregate $u$ can have wildly varying scale depending on which experts were picked and their weights. K3 inserts an RMSNorm before the up-projection. RMSNorm (Zhang & Sennrich, 2019) is just: divide the vector by the root-mean-square of its entries, then scale by a learned gain. This one line materially improves validation loss.

6.3 SiTU-GLU — a bounded activation

The classic transformer FFN uses SwiGLU: $\text{SwiGLU}(x) = (x \cdot \sigma(W_g x)) \odot (W_u x)$. Both factors are unbounded, so if a "gate" and "up" projection happen to spike together on one channel, you get huge activations — which overflow in low-precision arithmetic (a real problem for MXFP4/FP8 training).

K3 introduces SiTU-GLU (Sigmoid Tanh Unit GLU), which "smoothly caps" both branches with a scaled tanh:

$$\text{SiTU-GLU}(x) \;=\; \beta_1 \tanh\!\left(\frac{W_g x}{\beta_1}\right) \odot \sigma(W_g x) \odot \beta_2 \tanh\!\left(\frac{W_u x}{\beta_2}\right)$$

with $\beta_1 = 4$ and $\beta_2 = 25$. Two nice properties:

• Near the origin, $\beta \tanh(z/\beta) \approx z$ — so SiTU behaves like SwiGLU on typical inputs.
• Far from the origin, both branches saturate. The output is bounded: $\|\text{SiTU-GLU}(x)\|_\infty \le \beta_1 \beta_2 = 100$.

6.4 Quantile Balancing — solving load balancing without an auxiliary loss

The problem. In MoE, some experts get overwhelmed with tokens ("hot"), others starve ("dying"). With 896 experts, this is a serious production risk: hot experts slow down training, dying experts waste parameters.

Prior state of the art (DeepSeek-V3) added a per-expert bias $b_j$ to the router score, and nudged $b_j$ by $\pm \gamma$ at every step depending on whether expert $j$ was over- or under-utilized. Simple. But at 896 experts the fixed step size $\gamma$ becomes very hard to tune: too small → slow to adapt, too large → oscillates.

K3's idea. Solve for the bias exactly. Every step, compute the histogram of router scores per expert, and set the bias so that exactly the target number of tokens will route to each expert.

Concretely (Equation 14):

$$\hat b_j^{(t+1)} \;\leftarrow\; -\,\text{quantile}_{\,1 - k/n}\!\left(s_{:,j}^{(t)} - \alpha^{(t)}\right), \qquad b^{(t+1)} \;\leftarrow\; \hat b^{(t+1)} - \text{mean}(\hat b^{(t+1)})\, \mathbf 1,$$

where the "margin" $s_{i,j} - \alpha_i^{(t)}$ is the biased router score minus the "cutoff" the token needs to enter its top-$k$. The bias for expert $j$ is set to whatever value would put exactly $mk/n$ tokens above the cutoff — this is a quantile of the margin distribution.

Why it is practical. At scale, "computing the exact quantile" over billions of margins is impossible. K3 uses a histogram approximation: each rank bins its local margins into $\sim 8000$ buckets per expert, a single all-reduce sums the bin counts, and the quantile is read off the pooled histogram. Total extra cost: one small collective per step. No auxiliary loss, no tunable step size, and the bias is frozen at inference.

6.5 Per-Head Muon — one optimizer detail worth mentioning

Muon is a recent optimizer for hidden-layer weights (Jordan et al., 2024, scaled by Moonshot in 2025) that updates whole weight matrices with a specially-normalized step. K3 extends this to treat each attention head as its own optimization group, which stabilizes training when head widths differ or when some heads specialize faster than others.

---

7. Vision — MoonViT-V2

Kimi K3 is natively multimodal — vision is not bolted on after training. Images and video frames go through MoonViT-V2, a vision transformer trained jointly with the language model. A small MLP projector maps its visual features into the same embedding space the text tokens live in.

🖼️

Pixels

image or video frame

MoonViT-V2

patch → tokens

MLP proj

align to text space

Embedding

unified sequence

K3 backbone

1. 1
   Pixels

   image or video frame
2. 2
   MoonViT-V2

   patch → tokens
3. 3
   MLP proj

   align to text space
4. 4
   Embedding

   unified sequence
5. 5
   K3 backbone

Practical consequence: the model can, mid-reasoning, execute code to zoom or crop an image, feed the result back as a new visual observation, and continue — the "vision-in-the-loop" pattern the blog highlights for game-dev and CAD tasks.

---

8. Training — what actually happened

8.1 Pre-training

• Scale. 2.8T parameters, 104B active, MoE sparsity ≈ 27.
• Data. Refined recipes over Kimi K2; the report emphasizes better data curation contributed to the 2.5× win alongside architecture.
• Efficiency. Kernel-level co-design: fused kernels for KDA, sequence parallelism for AttnRes, custom kernels for the latent MoE.

8.2 Post-training pipeline

📚

SFT

supervised fine-tune on curated demos

Domain RL

general tasks / general agents / coding agents, each at low / high / max effort

Multi-teacher distillation

consolidate specialists into one unified model

1. 1
   SFT

   supervised fine-tune on curated demos
2. 2
   Domain RL

   general tasks / general agents / coding agents, each at low / high / max effort
3. 3
   Multi-teacher distillation

   consolidate specialists into one unified model

8.3 Reinforcement Learning across "effort levels"

The team trains the model at multiple reasoning-effort levels ("low", "high", "max") within each domain, then merges them. This is what lets the same K3 model spend a small budget on an easy question or a much larger budget on a difficult agentic task — without regressing on either.

8.4 On-policy distillation with dense rewards

Instead of the usual "sparse verifier" reward, K3 uses a dense token-level KL-style reward comparing the student to a domain-specialist teacher:

$$r_{\text{opd}}(y_t \mid e, x, y_{<t}) \;=\; \text{clip}\!\left(\text{sg}\log\frac{\pi_{\text{teacher}}^{(d,e)}(y_t)}{\pi_\theta(y_t)},\ -R_{\max},\ R_{\max}\right).$$

Every symbol: $e$ = effort level, $d$ = domain, $\text{sg}$ = stop gradient, $R_{\max}$ = clipping threshold to prevent extreme advantages. Every generated token gets a reward — much denser signal than "success/failure at the end of a trajectory," which makes long-horizon, tool-using trajectories easier to train.

8.5 Unified white-box RL environment

Every "agent harness" (Kimi Code, Claude Code, Codex, OpenClaw, Hermes) is decomposed into interchangeable modules: tools, prompts, context management, memory, subagents. During RL, tasks are matched with randomized harness configurations. This prevents overfitting to any single harness's conventions — the resulting policy generalizes to your harness.

8.6 Knowledge-graph-guided task synthesis

Post-training tasks are only as good as their source material. Moonshot builds a self-evolving, hierarchical knowledge graph (agents recursively explore and add nodes), then samples related nodes as keyword sets to guide web retrieval and task synthesis.

8.7 Why nine teachers exist

K3 does not train one specialist for every benchmark. It defines three broad RL domains:

1. General tasks — knowledge, reasoning, vision, faithfulness, search, and professional work.
2. General agents — long-horizon assistance, deep research, and extended writing.
3. Coding agents — software engineering, kernels, coding experience, and web development.

Each domain is trained at three reasoning-effort levels: low, high, and max. The Cartesian product gives $3\times3=9$ expert policies. Multi-Teacher On-Policy Distillation then consolidates them into one deployed policy conditioned on the requested effort.

This separation addresses a real optimization conflict. A policy rewarded only for maximum task success tends to spend too many tokens on easy problems. A policy rewarded only for brevity gives up too early on hard problems. K3 trains the effort points explicitly rather than hoping one reward weight produces all three behaviors.

8.8 Per-problem reasoning budgets

Let $b_0(x)$ be an initial token budget estimated for problem $x$, and let $T(y)$ be the token cost of trajectory $y$. K3 applies a budget multiplier $\tau$ and overrides the task reward with $-1$ when

$$T(y) > \tau b_0(x).$$

For a general reasoning task, $T(y)$ counts thinking tokens. For an agentic task it counts cumulative output, including reasoning traces and tool-call arguments. Training begins with a larger $\tau$ for the max-effort policy, then anneals $\tau$ to derive high- and low-effort experts.

The important idea is relative budgeting. A fixed 8,000-token cap treats a one-line arithmetic problem and a repository-wide refactor identically. Scaling the budget from a problem-specific baseline creates a more meaningful pressure toward efficiency.

8.9 Partial rollouts and the straggler problem

Suppose training samples $K$ completions for each of $N$ prompts, so $NK$ trajectories run concurrently. Tool-using trajectories have heavy-tailed duration: one may finish in seconds while another waits on a build, browser, or sandbox for minutes. A synchronous trainer that waits for the slowest trajectory leaves expensive accelerators idle.

K3 pauses generation after a fraction $\lambda\in(0,1)$ of trajectories complete:

$$N_{\text{ready}} = \lambda NK.$$

Optimization starts immediately on the completed work. Paused trajectories are queued and prioritized for resumption in the next iteration. Once all $K$ responses for one prompt finish, that group becomes ready for policy optimization.

The cost is policy staleness: one long trajectory can span several optimizer updates, so later tokens were sampled by an older policy. K3’s optimizer uses per-token regularization to keep updates in a localized neighborhood, making the extreme off-policy mixture tolerable.

8.10 Reward models, rubrics, and verbosity hacking

Many coding and mathematics tasks have executable verifiers. Open-ended research and professional writing do not. K3 uses an Agentic Generative Reward Model for these tasks. The judge must follow a protocol:

1. Read the outcome or artifact.
2. Generate a task-specific rubric.
3. Score each candidate against that rubric.
4. Record the rubric scores in a scorepad.

The protocol makes the basis of comparison explicit. It does not eliminate judge error, but it is more auditable than an unexplained scalar.

Longer answers can appear more thorough and exploit a judge’s preferences. To reduce that reward hack, K3 estimates an initial verbosity $\ell_0$, chooses multiplier $\sigma$, and automatically loses a comparison when output length exceeds $\sigma\ell_0$. This is the output-length analogue of reasoning-budget control.

8.11 Multi-Teacher On-Policy Distillation, line by line

For domain $d$, effort $e$, query $x$, prefix $y_{<t}$, sampled token $y_t$, teacher $\pi_{\text{teacher}}^{(d,e)}$, and student $\pi_\theta$, K3 defines

$$r_{\text{OPD}}^d(y_t\mid e,x,y_{<t})= \operatorname{clip}\left( \operatorname{sg}\left[ \log\frac{\pi_{\text{teacher}}^{(d,e)}(y_t\mid x,y_{<t})} {\pi_\theta(y_t\mid e,x,y_{<t})} \right],-R_{\max},R_{\max}\right).$$

Interpretation:

• If the teacher assigns the sampled token more probability than the student, the log-ratio is positive.
• If the student already overweights it, the ratio is negative.
• sg means stop gradient: treat the reward number as fixed rather than differentiating through its construction.
• Clipping to $[-R_{\max},R_{\max}]$ prevents rare tiny probabilities from producing enormous, destabilizing advantages.
• Because this reward exists at every generated token, it gives denser credit than one terminal pass/fail signal.

The report also states a useful negative result: more fine-grained top-$k$ distillation objectives did not clearly improve convergence or final performance in their setting. Reporting failed refinements matters; it prevents the architecture story from becoming survivor-biased mythology.

8.12 Deployment-aware learning starts before deployment

K3 applies quantization-aware training through SFT and RL, not only after alignment. Routed expert weights use MXFP4 and their input activations use MXFP8, while attention projections, latent MoE projections, routers, and shared experts remain at higher precision.

That alignment between rollout precision and serving precision removes a common mismatch. If RL explores with high-precision weights but production serves a separately quantized approximation, the deployed policy can encounter action probabilities it never optimized under. K3 makes quantization noise part of the policy’s training environment.

---

9. Infrastructure — how they actually ran this

The tech report spends the most pages here, because at 2.8T parameters, infrastructure is the model.

9.1 MoonEP — perfectly balanced expert parallelism

Expert parallelism means: different GPUs hold different experts. If token routing is imbalanced, some GPUs sit idle. MoonEP enforces static computation shapes (each GPU processes the same fixed number of tokens per step) with zero-copy communication. Combined with Quantile Balancing, this eliminates the biggest source of MoE training stalls.

9.2 KDA Context Parallelism

For 1M-token sequences, no single GPU can hold the whole context. Context parallelism shards the sequence across GPUs and communicates the KDA recurrent state at chunk boundaries. K3 pairs this with state-aware prefix caching so that if two requests share a long prefix (very common in coding sessions), the KDA state at the shared boundary can be restored from cache instead of recomputed.

9.3 KDA-aware prefix cache — the deployment story

Normal LLM serving caches the "KV cache" of long prefixes so a second request with the same prefix skips prefill. For a 1M-token coding session with a 400K-token prefix, this is life-or-death for latency.

K3 has two caches per block: the MLA KV cache (grows per token) and the KDA recurrent state (fixed size, one per request). A cached prefix is only reusable if both are restorable at the same boundary. K3’s design:

Consequence: the announced API pricing — $0.30/MTok cache-hit input** vs. **$3.00/MTok cache-miss input — is a direct consequence of this design. Coding workloads hit the cache >90% of the time.

9.4 AgentENV — resumable microVM sandboxes for agentic RL

RL for a coding agent needs an environment where the agent can safely run arbitrary code. Containers are cheap but leak; VMs are safe but slow. AgentENV uses Firecracker microVMs with incremental checkpointing:

• Checkpoint latency: 133 ms
• Resume latency: 49 ms
• Pause/Resume: paused sandboxes cost 0 CPU/memory
• Fork: create a side-effect-free copy for reward judging
• Snapshot: for recovery

During K3 training and evaluation, 51.2 million sandboxes were created across 1.5 million distinct images. AgentENV is open-sourced here.

9.5 Fleet-level scheduling

Two policies keep production serving predictable:

• Cache-aware affinity scheduling. Each session pinned to the cluster holding its cache (via consistent hashing), with a hot-standby cluster for failover.
• Budget-based admission control. Each request class (short vs. million-token) gets its own resource budget so bursty long-context traffic can't starve short requests.

---

10. Deployment — how it stays cheap

10.1 MXFP4 quantization-aware training

MXFP4 is a 4-bit floating point format (Rouhani et al., 2023). K3 quantizes MoE expert weights (which dominate memory) to MXFP4, with activations at MXFP8, keeping attention and shared experts in higher precision. Quantization-Aware Training (QAT) is used throughout SFT and RL — so the model is trained with the exact quantization scheme it will be served under. No train/serve mismatch.

L3

Attention projections, MLA, router, shared experts

BF16 / FP16

L2

MoE expert activations

MXFP8

L1

MoE expert weights

MXFP4 (4-bit)

core

10.2 EAGLE-3 speculative decoding

K3's multi-token-prediction (MTP) head is fine-tuned into an EAGLE-3 draft model: a small network that predicts the next 7 tokens speculatively, which the full model then verifies in parallel. Instead of the standard KL loss, the draft is trained directly on the LK loss:

$$\mathcal L_{LK} \;=\; -\log \sum_{x \in V} \min\!\bigl(p(x),\, q(x)\bigr)$$

which is the negative log of the acceptance rate itself. This gives a measurable throughput win at inference.

10.3 KDA state replay for speculative decoding

A subtle problem: if the draft's 7 tokens include some that are rejected, the KDA state has already advanced. Rolling back a large state per step is expensive. K3 caches only the projected inputs of drafted tokens (much smaller than the state), then replays the accepted prefix on-chip in a single fused kernel. This is the concurrent-work "ReplaySSM" idea, integrated cleanly into the serving stack.

---

11. Benchmarks — what they mean

Take these with the usual grain of salt (max reasoning effort, harness-dependent), but Moonshot at least documented the harness they used per benchmark in the blog’s footnotes.

How to read each benchmark:

• DeepSWE — software-engineering issues from real GitHub repos; the model is scored on whether its patch passes tests.
• Terminal-Bench 2.1 — realistic command-line tasks; the harness runs commands and grades outcomes.
• FrontierSWE / SWE-Marathon / KCB 2.0 / ProgramBench — variations on "solve real bugs / write real code."
• BrowseComp — an agent has to browse the web and produce a verifiable answer. K3 is #1 here (with context compaction at 300K).
• AutomationBench / JobBench / AA-Briefcase — human-graded agentic knowledge-work benchmarks (finance, legal, ops).
• CharXiv / ZeroBench / MMMU-Pro / PerceptionBench — vision-heavy reasoning.

The pattern to notice: K3 wins on long-horizon agentic tasks (BrowseComp, AutomationBench, SWE-Marathon) — exactly where its 1M context, KDA cache-hit economics, and RL environments are designed to help.

---

12. Case studies (from the blog)

12.1 MiniTriton — a GPU compiler K3 wrote

Given a spec, K3 built MiniTriton: a Triton-like DSL frontend, MLIR-based IR passes, and a PTX code-generation pipeline, plus a PyTorch-like tensor library with autograd and NCCL primitives. Result: on an NVIDIA L20, MiniTriton beats torch.compile and Triton on geometric mean over its benchmark suite, approaches cuBLAS at large matmul sizes, and trains a GPT model end-to-end with a loss curve nearly identical to PyTorch reference (fp32 rounding-level agreement). Repo: github.com/MoonshotAI/minitriton.

12.2 nano-KPU — a chip K3 designed in 48 hours

K3 designed an inference-chip prototype for a "nano" model built on the same architecture (KDA + NoPE-MLA + Block AttnRes + INT4 MoE). Using open-source EDA tools on the Nangate45 standard-cell library, it produced a 4 mm² design that closes timing at 100 MHz and hits >8,700 tokens/sec decode throughput in RTL simulation. Repo: github.com/MoonshotAI/nano-kpu.

12.3 I–Love–Q astrophysics reproduction

To reproduce a classical result in computational astrophysics (universal relations between neutron-star moment of inertia, tidal Love number, and quadrupole moment), K3 read 20+ papers, implemented the full numerical pipeline, evaluated 300+ equations of state, spotted inconsistencies in published formulas, and produced an interactive HTML dashboard — in ~2 hours vs. weeks for a human researcher.

12.4 GWTC-5 gravitational-wave analysis

391 events analyzed in parallel using 20+ concurrent subagents, producing 7 figures, 2 tables, and a synthesis over 10+ papers.

12.5 Video editing from 56 clips

K3 cut its own teaser video: clip selection, motion-matched cuts, beat-synced timing, audio processing, multiple revisions. Because the model is natively multimodal, "understand the clip" and "write the edit decision list" happen in the same forward pass.

---

13. Limitations (from the blog)

L4

Sensitive to thinking history

if the harness drops <think> traces or you swap models mid-session, quality degrades sharply

L3

Excessive proactiveness

on ambiguous requests it may 'just decide' — needs explicit boundaries in AGENTS.md

L2

UX gap vs Fable 5 / GPT-5.6 Sol

raw capability is competitive; polish and refusal behavior lag

L1

Still trails top proprietary models overall

on the hardest coding + reasoning benchmarks

core

The first two are worth reading twice. K3 was trained in preserved-thinking-history mode, which means the RL policy assumes its own reasoning traces come back in the next turn. Standard OpenAI-style APIs that drop <think> blocks between turns can break it. Moonshot recommends the Kimi Code harness for that reason.

---

14. What this actually means for open source

Three things are new about K3:

1. A 3T-class model exists in the open. No open lab had crossed 2T before. K3 pushes the ceiling by ~40%.
2. The scaling advances are architectural, not just "more data / more compute." KDA + AttnRes + Stable LatentMoE are the design of the model, not tricks bolted on.
3. Infrastructure was co-designed and mostly open-sourced. FlashKDA, MiniTriton, nano-KPU, and AgentENV all have public repos. Anyone rebuilding a frontier model can start from these instead of reinventing them.

The gap to Fable 5 / GPT-5.6 Sol is real, but it is a capability gap, not an architectural gap. That is a very different position for open source than "we're one training run behind" — because now the training tools themselves are open.

---

15. Glossary (skim this after)

L16

MoE (Mixture of Experts)

many small FFNs; a router picks which few to run per token

L15

Sparsity ratio

total experts / active experts (K3 = 896/16 ≈ 56)

L14

Linear attention

O(L) attention via a recurrent state; loses precise recall

L13

Delta rule

error-correcting recurrent update; erase-then-write

L12

KDA

delta rule + channel-wise forget gate + bounded log-decay

L11

MLA

Multi-head Latent Attention — full attention with compressed KV cache

L10

AttnRes

learned attention across block outputs — non-sequential residuals

L9

LatentMoE

shared W↓ / W↑ let routed experts run in a compact latent space

L8

SiTU-GLU

smoothly-capped SwiGLU; bounded output for low-precision safety

L7

Quantile Balancing

exact per-expert bias from margin quantiles

L6

Per-Head Muon

Muon optimizer applied independently per attention head

L5

MXFP4 / MXFP8

block-scaled 4- and 8-bit floating point formats

L4

QAT

quantization-aware training — train under serving precision

L3

EAGLE-3

draft-and-verify speculative decoding, trained with LK loss

L2

MoonEP

expert-parallel training with static shapes, zero-copy

L1

AgentENV

resumable microVM sandbox for agentic RL

core

---

16. Complete YouTube course — zero background to K3 internals

The original version of this post contained no direct YouTube links, so there was nothing to remove or replace. The curriculum below is additive. Every link was checked through YouTube’s public oEmbed endpoint before publication; dead, private, restricted, duplicate, and hype-only videos were excluded.

There is not yet a rigorous official Kimi K3 video lecture that covers the technical report end to end. Use the Moonshot report for K3-specific facts and use these videos to master the prerequisite ideas the report assumes.

Tier 0 — first understand AI and language models

1. Large Language Models explained briefly — 3Blue1Brown — the quickest visual orientation.
2. AI, Machine Learning, Deep Learning and Generative AI Explained — IBM Technology — separates terms that are often used interchangeably.
3. Intro to Large Language Models (1-hour talk) — Andrej Karpathy — the best single broad introduction.
4. How I use LLMs — Andrej Karpathy — connects the architecture to practical use.

Checkpoint: You should be able to explain the difference between AI, machine learning, deep learning, generative AI, a language model, and an agent.

Tier 1 — vectors, matrices, probability, and loss

1. Vectors — Essence of linear algebra, Chapter 1 — 3Blue1Brown
2. Linear transformations and matrices — Chapter 3 — 3Blue1Brown
3. Matrix multiplication as composition — Chapter 4 — 3Blue1Brown
4. Eigenvectors and eigenvalues — Chapter 14 — 3Blue1Brown — optional, but useful for low-rank and conditioning discussions.
5. Neural Networks Part 5: ArgMax and SoftMax — StatQuest
6. Neural Networks Part 6: Cross Entropy — StatQuest

Checkpoint: Given $y=Wx$, identify the shapes of $x$, $W$, and $y$; explain dot product versus outer product; turn logits into probabilities; and explain why $-\log p_y$ punishes confident mistakes.

Tier 2 — how neural networks learn

1. But what is a neural network? — 3Blue1Brown
2. Gradient descent, how neural networks learn — 3Blue1Brown
3. Backpropagation, intuitively — 3Blue1Brown
4. Backpropagation calculus — 3Blue1Brown
5. Gradient Descent, Step-by-Step — StatQuest — slower alternative.
6. Neural Networks Part 2: Backpropagation Main Ideas — StatQuest — slower alternative.
7. MIT Introduction to Deep Learning 2025 — 6.S191 — university-level consolidation.

Checkpoint: Explain a forward pass, loss, gradient, chain rule, backward pass, learning rate, optimizer step, and why exploding activations destabilize training.

Tier 3 — Transformers and attention

1. Transformers, the tech behind LLMs — 3Blue1Brown
2. Attention in transformers, step-by-step — 3Blue1Brown
3. How might LLMs store facts — 3Blue1Brown — especially useful before MoE because it explains the FFN block experts replace.
4. Transformer Neural Networks, ChatGPT’s foundation, Clearly Explained — StatQuest
5. What are Transformers? — IBM Technology
6. AI Language Models & Transformers — Computerphile
7. Let’s build GPT from scratch — Andrej Karpathy — code-level capstone.
8. Stanford CS336 Lecture 1: Overview and Tokenization — advanced course entry point.

Checkpoint: Derive scaled dot-product attention, state the shapes of Q/K/V, explain causal masking, identify attention versus FFN computation, and explain why full attention is quadratic in sequence length.

Tier 4 — MoE, MLA, long context, RL, and efficient inference

1. Mixture of Experts: How LLMs get bigger without getting slower — Julia Turc
2. How DeepSeek Rewrote the Transformer — MLA — Welch Labs
3. How KV Cache Works — Toni
4. Reinforcement Learning from Human Feedback: From Zero to ChatGPT — Hugging Face
5. RLHF explained with math derivations and PyTorch code — Umar Jamil
6. State of GPT — Andrej Karpathy
7. Deep Dive into LLMs like ChatGPT — Andrej Karpathy
8. Speculative Decoding: How a Draft Model Makes LLMs Faster — Devsplainers
9. LLM Quantization Explained — Joydeep Bhattacharjee
10. Mamba and S4 Explained: Architecture, Parallel Scan, Kernel Fusion — Umar Jamil — useful comparison for recurrent long-context designs.

Checkpoint: Explain top-k expert routing, load imbalance, KV-cache growth, latent KV compression, policy/rollout/reward, quantization-aware training, and draft-then-verify decoding.

Tier 5 — vision and multimodal models

1. An Image is Worth 16×16 Words: ViT paper explained — Yannic Kilcher
2. Coding a Multimodal Vision-Language Model from scratch in PyTorch — Umar Jamil
3. CS231n Lecture 10: RNNs, Image Captioning, LSTM — Andrej Karpathy / Stanford — optional historical context.

Checkpoint: Explain image patches, a vision encoder, projector, visual tokens, spatial versus temporal attention, and why native multimodal pre-training differs from attaching a frozen encoder later.

A practical four-week schedule

• Week 1: Tier 0 + Tier 1. Rework the shape trace in §2.13 by hand.
• Week 2: Tier 2 + the first six videos in Tier 3. Build one tiny neural network or follow micrograd.
• Week 3: Finish Tier 3, then read §§3–6 of this post while watching Tier 4 videos 1–3.
• Week 4: Tier 4 videos 4–10 + Tier 5. Read the K3 technical report’s architecture, post-training, and infrastructure sections beside this post.

Do not try to memorize every acronym. At the end of each tier, explain the checkpoint aloud without notes. If you cannot, rewatch only the segment tied to the missing concept.

---

17. Reading path if you want to go deeper

1. The blog post first. kimi.com/blog/kimi-k3 has the images and case-study videos we couldn't inline here.
2. The tech report, Section 2 (architecture) — with this post open in a second tab, every equation should now be readable.
3. Kimi Linear paper (arXiv 2510.26692) — the original KDA introduction.
4. DeepSeek-V3 tech report (arXiv 2412.19437) — the auxiliary-loss-free routing K3 improves on, and the MLA baseline.
5. LatentMoE (arXiv 2601.18089) — the paper Stable LatentMoE builds on.
6. Muon optimizer (Jordan et al., 2024) — for Per-Head Muon.
7. EAGLE-3 (arXiv 2503.01840) — for the draft-model design.

---

Appendix — A note on how this post was formatted (reusable guideline)

For future deep-dive posts, this is the format that worked. Keep it.

L10

Sources block at the very top

primary blog + tech report + repos; readers can verify anything you say

L9

One-page TL;DR + headline chart

so a busy reader gets 80% in 60 seconds

L8

Background section (skippable)

for readers who don't have the prereqs — mark as skippable

L7

One diagram per concept

use ArchitectureDiagram / DataFlow / ConceptLayers / XYPlot — no ad-hoc SVGs

L6

Every equation with every symbol defined

no orphan variables, no `where obvious'

L5

Story mode → Diagram mode → Technical mode

three passes per idea, in that order

core

L4

Track color consistency

one track (violet=LLM here) throughout for visual language

L3

Glossary at the end

single skim reference for the terms introduced

L2

Reading path for deeper study

primary sources ordered by prerequisite

L1

MDX escape hygiene

no `<digit`, no bare `{...}` in prose; JSX attrs use `&apos;` or `&quot;` entities

The ' / " rule is not stylistic — backslash escapes inside JSX attributes silently drop the attribute and desync neighboring diagrams. Every future post should follow this or the diagrams will render blank without a build error.