Kimi K3: architecture, derivations and a pinned-source audit

Kimi K3 From Zero to Deep — Math, Architecture, Training, and Systems

Sources (please open these too):

• Moonshot AI blog: kimi.com/blog/kimi-k3
• Technical report (PDF): github.com/MoonshotAI/Kimi-K3
• Model weights: huggingface.co/moonshotai/Kimi-K3
• Related open-source projects referenced: FlashKDA, MiniTriton, nano-KPU, AgentENV
• All figures/images referenced in this post live in the original blog page — open the Moonshot blog in a second tab to see them next to each explanation.

Evidence boundary. Kimi-specific architecture, training and systems descriptions below are attributed to the publisher’s report and model summary at commit 3cb39dfd32e51c3328e2e4b4af21341247d06c43, checked in the 8 October 2026 pinned-source audit and reused here for the same revision. Case studies use the publisher’s blog snapshot captured on 8 October, not independently reproduced achievements. Repository identity and the publisher’s model-card release description were freshly checked on 9 October 2026. Local checks reproduce only named small arithmetic/recurrence examples, not the model, deployment or research results. Optional video links were not independently revalidated in this correction.

Identity and date. The official repository main endpoint still returned commit 3cb39dfd32e51c3328e2e4b4af21341247d06c43 (committer date 2026-08-06) when checked on 2026-10-09. That timestamp identifies a repository revision, not an independently established model-publication date. This lesson describes that publisher-labelled Kimi K3 revision, not a guessed future model, not every current Kimi product setting and not the identity of whichever assistant a reader is using. The available report supports architecture and attributed experiments; weights, licensing completeness and independent capability replication remain unverified here.

One-line summary. Moonshot AI describes Kimi K3 as a 2.8-trillion-parameter Mixture-of-Experts language model with native vision and a 1-million-token context window, released as open weights under its named Kimi K3 License. This is the publisher’s release description, not an independent check of every weight file or permission for a particular use. Its main technical contributions are adaptations of recurrent attention (Kimi Delta Attention), information routing across layers (Attention Residuals), and extreme-sparsity expert computation (Stable LatentMoE). The report attributes an approximately 2.5× scaling-efficiency comparison to the combined recipe, including data and training changes—not to a measured speedup from one component.

That sentence is dense. The rest of this post unpacks every term in it — starting from "what is attention" — so that by the end you can read the actual tech report and understand what each equation is doing and why.

0. How to read this post

This post is a layered explainer: start with the motivation, trace the shapes or diagrams, then work through the equations. Not every section needs a diagram or a separate box. If an equation is unfamiliar, use the background vocabulary first and return to the worked KDA state update; if you already know the basics, start with the architecture table and distinguish sequence, depth and expert routing.

1. The one-page TL;DR

The pinned evaluation compares multiple systems under task-specific harnesses and effort settings. Kimi K3 leads some listed agentic tasks and trails other systems on others; no universal model ranking follows. In particular, percentages, pass-at-k rates and Elo values cannot be averaged into one capability score.

Read these as reported results for the evaluated systems, not as proof that any one architectural choice caused a score. The next sections explain the mechanisms that could support these workloads; the later training and benchmark sections describe the conditions under which they were evaluated.

2. Background — what is a language model even doing?

Skip this section if you already know the basics of transformers.

2.1 Language modeling in one sentence

An autoregressive language model reads a token prefix and predicts a distribution for its next token. This is the autoregressive base objective; post-training adds other objectives and environment feedback, so it is not the only objective in the full system.

Everything hard about scaling a language model is (a) how the "N transformer blocks" mix information along the sequence, and (b) how much useful compute you can pack into each block.

2.2 The three axes of scale

The Kimi team frames K3 as scaling three complementary dimensions:

1. Sequence: mix information across token positions using recurrent KDA and periodic global MLA.
2. Depth: select information from earlier depth groups with Block AttnRes.
3. Width/capacity: store many expert functions while routing each token to a subset through a compact latent path.

These are distinct axes, not three interchangeable parameter counts. Keep this triangle in mind. Every architectural choice below is aimed at one of these three axes.

2.3 Standard attention, in one paragraph

Given a sequence of token vectors, self-attention lets each token pull information from permitted positions by (i) projecting each vector into three roles — Query (Q), Key (K), Value (V), (ii) computing a similarity score between each Query and every Key, (iii) softmax-normalizing those scores, and (iv) using them as weights to average the Values.

For causal language modeling, $M_{ij}=0$ when $j\le i$ and $M_{ij}=-\infty$ when $j>i$, so softmax assigns zero probability to future positions. The state at position $i$ can read its own input token while predicting the next one. Mask before softmax; zeroing future weights afterward without renormalizing changes the output scale. In the shape trace in §2.13, apply this mask before step 4.

The $1/\sqrt{d_k}$ scale has a simple motivation: if query and key coordinates are independent, zero-mean and unit-variance, their dot product has variance $d_k$. Dividing by $\sqrt{d_k}$ gives unit variance under those assumptions and reduces softmax saturation as width grows. Learned activations need not obey those assumptions exactly; this is the initialization-scale argument, not a universal statistical guarantee.

The problem: dense pairwise attention has quadratic arithmetic in sequence length; a materialized score matrix also has quadratic memory, while tiled exact attention can avoid materializing it. At $L = 1{,}000{,}000$ this is a trillion conceptual query–key pairs per layer per head. This makes a naive materialized implementation impractical for many hardware budgets. Tiling changes memory cost and distributed execution changes placement; the hybrid design changes the mix of recurrent and pairwise work, not the existence of every quadratic operation.

2.4 Tokens, vocabulary, and embeddings

A model does not read characters or words directly. A tokenizer converts text into integer IDs from a fixed vocabulary. A token may be a whole word, part of a word, punctuation, whitespace, or a byte sequence. K3 uses a vocabulary of roughly 160,000 entries. The same sentence can therefore occupy different numbers of tokens under different tokenizers.

If token ID $i$ enters the model, an embedding table $E \in \mathbb{R}^{|V| \times d}$ selects row $E_i \in \mathbb{R}^{d}$. That row is not a definition of the token. It is a learned coordinate: a point in a high-dimensional space whose geometry becomes useful during training.

Three terms that are easy to confuse:

• Token — one discrete vocabulary item.
• Embedding — the dense vector representing that token at the model entrance.
• Hidden state — the evolving vector for that token after one or more layers have mixed information into it.

A hidden width of $d=7168$ means every textual position is represented by 7,168 floating-point coordinates inside K3. The model repeatedly transforms those coordinates; it does not repeatedly manipulate the original text.

Watch first: Let’s build the GPT Tokenizer — Andrej Karpathy. The tokenizer is outside the neural network, but every context-length, cost, and latency number is measured in its output tokens.

2.5 The minimum linear algebra

You only need four objects to read most of the K3 equations:

A matrix multiplication $y=Wx$ is a learned change of coordinates. Each output coordinate is a weighted sum of all input coordinates:

The dot product $q^\top k=\sum_j q_jk_j$ combines vector norms and angular alignment. For nonzero vectors its sign distinguishes an acute from an obtuse angle, but a large value can come from large norms rather than a small angle. Attention turns these alignments into retrieval weights.

The outer product $kv^\top$ is different. If $k \in \mathbb{R}^{d_k}$ and $v \in \mathbb{R}^{d_v}$, then $kv^\top \in \mathbb{R}^{d_k \times d_v}$. It creates a matrix that associates a key direction with a value. KDA writes exactly this object into its recurrent memory.

The identity matrix $I$ leaves a vector unchanged. Therefore $I-\beta kk^\top$ means “keep everything except some component along $k$.” That is why it acts as KDA’s erase operator.

A diagonal matrix $\operatorname{Diag}(\alpha)$ scales each coordinate independently. KDA’s channel-wise forget gate uses it to retain some memory channels longer than others.

Watch visually:

1. But what is a neural network? — 3Blue1Brown
2. Gradient descent, how neural networks learn — 3Blue1Brown
3. Backpropagation calculus — 3Blue1Brown

2.6 Probability, logits, softmax, and temperature

The model ends each step with one unnormalized score, called a logit, for every vocabulary token. If $z_i$ is token $i$’s logit, softmax converts all logits into a probability distribution:

• $p_i \ge 0$ and $\sum_i p_i=1$.
• $T>0$ is temperature. Smaller positive $T$ sharpens the distribution; larger $T$ flattens it. Greedy decoding is a separate limiting decision, not division by zero.
• Adding the same constant to every logit changes no probability. Implementations exploit this by subtracting $\max_j z_j$ before exponentiating, preventing overflow.

The same softmax idea appears twice in this post:

1. Vocabulary softmax chooses the next token.
2. Attention softmax chooses how much information to retrieve from each position or depth block.

A categorical distribution is simply a probability over discrete choices. Sampling from it creates variation; choosing $\arg\max_i p_i$ is deterministic greedy decoding.

2.7 Loss, information, entropy, and perplexity

For a training example whose correct next token is $y$, the negative log-likelihood or cross-entropy loss is

This is measured in nats when the logarithm is natural. It assigns a small penalty to a confident correct prediction and an enormous penalty to a confident wrong prediction. Averaging it over tokens gives the language-model training loss.

Entropy measures uncertainty inside a distribution:

A distribution concentrated on one token has low entropy; a flat distribution has high entropy. Perplexity is $\exp(\text{average loss})$. It can be read loosely as the model’s effective number of plausible next-token choices, but only when comparing the same tokenizer and data distribution.

KL divergence measures how differently two distributions assign probability:

K3’s multi-teacher distillation uses a clipped token-level log-ratio related to this quantity. The ratio tells the student whether its probability for the sampled token is below or above the appropriate expert teacher’s probability.

2.8 Gradients, backpropagation, and optimizers

A gradient $\nabla_\theta \mathcal{L}$ is the vector of partial derivatives of the loss with respect to every parameter $\theta$. It points toward the direction of fastest local increase, so gradient descent moves the other way:

where $\eta$ is the learning rate. Backpropagation applies the chain rule from the loss backward through every operation to compute those derivatives efficiently.

Real optimizers add memory and normalization:

• Momentum averages recent gradients, reducing noisy zig-zag motion.
• Adam/AdamW track per-parameter first and second moments and scale coordinates independently.
• Muon orthogonalizes matrix-valued momentum updates so singular directions receive a better-balanced update.
• Per-Head Muon, K3’s refinement, performs that orthogonalization separately for each attention head instead of coupling all heads in one matrix.

The scientific idea is conditioning. If a loss surface is extremely steep in one direction and flat in another, one global step size performs badly. Normalization and optimizer geometry make progress less sensitive to that mismatch.

Code-level foundation: The spelled-out intro to neural networks and backpropagation: building micrograd — Andrej Karpathy.

2.9 Neural-network anatomy: attention, FFN, residual, normalization

A Transformer layer alternates two kinds of computation:

1. Attention mixes information across token positions. It answers “which earlier positions should this token read?”
2. The feed-forward network (FFN) mixes information across channels within each token. It answers “what nonlinear features should this token compute after reading?”

Each sublayer is wrapped by a residual connection. Instead of replacing $x$ with $f(x)$, it forms roughly $x+f(x)$. This creates a direct path for information and gradients through depth.

Normalization controls scale. RMSNorm divides by the root mean square of the coordinates:

where $g$ is learned and $\epsilon$ prevents division by zero. Unlike LayerNorm, RMSNorm does not subtract the mean.

A head is one independent attention subspace. Multiple heads let the layer represent several retrieval patterns at once. “96 attention heads” does not mean 96 separate models; they are parallel slices inside one layer.

Best conceptual pair:

• Transformers, the tech behind LLMs — 3Blue1Brown
• Attention in transformers, step-by-step — 3Blue1Brown

2.10 Dense models, sparse models, and Mixture-of-Experts

A dense feed-forward layer applies its selected weight matrices to every token instead of routing tokens among alternative expert networks. That does not mean every stored parameter is accessed: input embeddings select rows, and implementation details affect actual traffic. A Mixture-of-Experts layer keeps many alternative FFNs called experts and uses a router to select only a few for each token.

If a layer has $N$ routed experts and chooses top-$k$, then its expert activation fraction is $k/N$. K3 chooses 16 of 896 routed experts, so only about 1.8% of routed experts execute for one token. Shared experts still run for every token.

This explains why three parameter counts can all be valid:

• Total parameters describe storage and potential capacity.
• Activated parameters describe roughly how much of the model participates per token.
• FLOPs per token describe arithmetic, which also depends on sequence length, attention, batch shape, and implementation.

MoE is not free. The router can overload popular experts; tokens and activations must move between GPUs; underused experts learn slowly. K3’s LatentMoE, Quantile Balancing, and MoonEP address those separate problems.

Watch: What are Mixture-of-Experts Models — Hugging Face.

2.11 Sequence complexity, memory, bandwidth, and precision

Big-O notation describes how cost grows, not the exact runtime. For sequence length $L$:

• Full attention evaluates an $L\times L$ pairwise pattern, so its pairwise arithmetic is $\mathcal{O}(L^2)$ for fixed head widths. A tiled implementation need not materialize the whole matrix.
• Recurrent or linear attention updates fixed-size state once per token, so sequence scaling is $\mathcal{O}(L)$.
• A KV cache grows with generated context because full-attention layers retain keys and values per token.
• KDA instead carries a fixed-size recurrent state per layer, though K3 still has periodic MLA layers with a compressed KV cache.

On modern accelerators, memory bandwidth often matters as much as arithmetic. Moving a weight from HBM can cost more time than multiplying it. MoE reduces arithmetic per token but introduces all-to-all communication. Quantization stores more weights per byte and moves less data.

Floating-point formats trade range and precision:

• FP32: wide and accurate, expensive.
• BF16/FP16: common training compute formats.
• FP8: fewer bits, higher throughput, tighter numerical constraints.
• MXFP4: block-scaled 4-bit format used for K3’s routed expert weights during deployment-aware post-training.

A block scale lets a small group of low-bit numbers share a higher-precision scale. Quantization-aware training simulates those rounding and saturation effects while learning, giving the model a chance to adapt instead of compressing it only after training.

2.12 Reinforcement learning vocabulary before K3’s post-training section

K3’s post-training report assumes a compact RL vocabulary:

• Policy $\pi_\theta$ — the model’s probability distribution over next actions or tokens.
• Trajectory / rollout — one complete interaction: prompts, reasoning, tool calls, observations, and answer.
• Reward — a scalar signal saying how desirable an outcome or action was.
• Verifier — program or judge that decides whether an output satisfies a criterion.
• On-policy — data sampled from the policy currently being optimized.
• Off-policy / stale data — data produced by an older policy.
• Credit assignment — deciding which earlier actions caused a later success or failure.
• Distillation — training a student to match one or more teacher distributions.
• Reward hacking — exploiting the reward rule without solving the intended task.

K3’s agent trajectories can span many tool calls and training iterations. Partial rollouts reduce waiting for stragglers, but create stale data. Token-level regularization constrains how far the policy moves, while budget penalties discourage unnecessary reasoning or verbosity.

Watch: Reinforcement Learning with Human Feedback, Clearly Explained — StatQuest. K3’s exact algorithm is broader than classic RLHF, but this establishes policy, reward model, preference comparison, and optimization.

2.13 A worked shape trace through one attention head

Suppose a toy batch has $B=2$ sequences, each $L=4$ tokens, hidden width $d=8$, and $H=2$ heads. Each head has $d_k=d_v=4$.

1. Input hidden states: $X \in \mathbb{R}^{2\times4\times8}$.
2. Linear projections: $Q,K,V \in \mathbb{R}^{2\times2\times4\times4}$ after splitting into heads.
3. Full-attention scores: $QK^\top \in \mathbb{R}^{2\times2\times4\times4}$.
4. Softmax runs across the final key-position axis.
5. Weighted values return $O \in \mathbb{R}^{2\times2\times4\times4}$.
6. Concatenate heads to recover $\mathbb{R}^{2\times4\times8}$.

At one million tokens, the score matrix for one sequence and one head would conceptually have $10^{12}$ entries. At two bytes each, merely storing that matrix would require about 2 TB before accounting for other heads, layers, gradients, or workspace. FlashAttention avoids materializing the full matrix, but it cannot remove the underlying quadratic pairwise computation. KDA changes the algorithm instead.

2.14 What “science” means in this report

K3 combines four kinds of scientific evidence:

1. Mathematical derivation — equations specify recurrences, losses, and communication operators.
2. Controlled ablation — change one component at smaller scale and compare validation loss or downstream metrics.
3. Scaling-law experiment — fit behavior across model size, token budget, batch size, and learning rate, then choose a large-run recipe.
4. Systems measurement — benchmark kernels, communication, cache hit behavior, sandbox latency, and end-to-end throughput.

None alone proves general intelligence. Benchmarks are empirical measurements under a particular harness, prompt, reasoning budget, tool set, and date. A score difference can come from the base model, the agent harness, context management, tools, or the evaluator. Read K3’s benchmark table as evidence about a complete evaluated system—not a timeless ranking of isolated neural networks.

3. Architecture overview

Kimi K3's block is unusual. The main repeating attention pattern contains three KDA layers followed by one Gated MLA layer, plus a final MLA layer, with Stable LatentMoE as the repeated sparse feed-forward design (the model summary separately lists one dense layer). Above that, Attention Residuals rewire how blocks talk to each other, and at the bottom MoonViT-V2 pipes vision into the same embedding space.

Follow the text and image paths into the shared representation, then distinguish three jobs: KDA and MLA mix information across token positions, the expert networks transform each token, and AttnRes routes information across depth. The diagram compresses repeated structure: the channel-mixing path is not limited to MLA layers. Use the shape table below for counts, and the next sections for how each mechanism works.

This is a mechanism map, not a literal graph of every operation. The official summary lists one dense layer; do not read the repeated MoE box as evidence that every feed-forward layer is sparse. The 3:1 attention repeats plus the final global-attention layer account for 93 layers, independently of the eight AttnRes depth groups.

3.1 Exact model shape — keep these counts separate

These numbers describe different axes. The 93 sequential layers are partitioned into eight depth groups for AttnRes. Within the layer stack, KDA is the dominant token-mixing mechanism and MLA appears periodically as a full-attention checkpoint. Do not multiply “eight groups” by “four layer types” and conclude the model has 32 layers—the diagram is conceptual, not a literal layer count.

The routed expert fraction is $16/896 \approx 1.79\%$ per token. That does not mean only 1.79% of the entire network runs: shared experts, attention, routers, embeddings, and other dense components also execute. The report’s rounded104B activated count summarizes the selected model parameters; it is not a FLOP count or memory-allocation plan.

3.2 Model-card arithmetic: two fractions that must not be confused

The selected routed-expert fraction is 16/896 = 1/56, approximately 1.79%. The active-to-total parameter ratio is 104/2800, approximately 3.71%. They differ because total active computation is not just the fraction of routed experts: shared and other model components contribute, and the headline counts are rounded.

from fractions import Fraction
from math import isclose
 
routed_fraction = Fraction(16, 896)
active_fraction = 104 / 2800
assert routed_fraction == Fraction(1, 56)
assert round(float(routed_fraction) * 100, 2) == 1.79
assert round(active_fraction * 100, 2) == 3.71
assert 69 + 24 == 93
assert 2**20 == 1_048_576
ideal_weight_bytes = 2.8e12 * 4 / 8
assert isclose(ideal_weight_bytes / 10**12, 1.4)
print(f"routed={float(routed_fraction):.2%}; active={active_fraction:.2%}")
print(f"idealized 4-bit weight payload={ideal_weight_bytes / 10**12:.1f} TB")

The 1.4 TB number assumes every one of 2.8 trillion parameters occupies exactly four bits. It is not an observed checkpoint size or a complete serving-memory estimate. Quantization scales, metadata, higher-precision components, activations, caches, routing buffers and runtime overhead add requirements. The report/model summary describes MXFP4 weights and MXFP8 activations, but that does not justify ignoring all other memory.

Nor does 104B active mean only 104B parameters need to be stored for arbitrary requests: different tokens can choose different experts. Expert placement, offloading and interconnect costs are systems choices. No GPU count or tokens-per-second promise follows from this worksheet alone.

The technical report Table 1 gives 2.78T total and 104.2B active parameters; the public summary rounds these to 2.8T and 104B. The public context specification is 1,048,576 tokens, not evidence of task accuracy across that length. Keep 93 layers distinct from 96 attention heads, and the two shared experts distinct from the 16 selected routed experts.

4. Kimi Delta Attention (KDA) — linear-time attention with a "forget gate"

Story mode. Standard attention is $\mathcal{O}(L^2)$. Linear attention is a family of tricks that reduces this to $\mathcal{O}(L)$ by rewriting attention as a recurrence: instead of comparing every token to every other token, you maintain a compact "state matrix" that summarizes everything you have seen so far, and you update it one token at a time. KDA is Moonshot's tuned version of one of these tricks, called the delta rule, plus a channel-wise forget gate.

If that felt fast, don't worry — we'll unpack every word.

4.1 Why linear attention exists at all

The classical result behind linear attention is that if you replace the $\text{softmax}$ in attention with a kernel $\phi$ that factorizes, you can rewrite

Read this slowly. The right-hand side says: to compute the output at token $t$, you don't need every previous key and value — you only need $S_t$, a $d_k \times d_v$ matrix that has been rolled forward one token at a time. Each new token adds $\phi(k_t)v_t^\top$ to $S_t$. This token-mixing operator admits a recurrent evaluation. It can still sit inside a Transformer-style hybrid block; “recurrent” describes the state update, not a prohibition on the surrounding architecture.

Cost: $\mathcal{O}(L \cdot d_k d_v)$ — linear in sequence length. Recurrent inference-state memory: constant in sequence length for this component. Training activations and the hybrid MLA cache are separate memory consumers.

The displayed unnormalized additive recurrence has no explicit erase or decay operation. Accumulated associations can interfere as context grows. Kernel-normalized linear attention also tracks a normalizer; the simplified equation above omits it deliberately and is not an exact softmax-attention identity.

4.2 The delta rule — writing over yourself instead of adding

The delta rule (Schlag et al., 2021) fixes this by making the recurrence look like an error-correcting write:

The scalar $\beta_t \in (0, 1)$ is a per-token "write strength." Read this as: "erase whatever the current key would have retrieved from the old state, then write in the new value." It behaves much more like a mutable dictionary and less like an infinite accumulator.

4.3 KDA's contribution — channel-wise forgetting

KDA (Kimi Team, Kimi Linear, 2025) adds one more piece: a channel-wise decay vector $\alpha_t \in (0, 1)^{d_k}$. Each channel (each row of the state) is multiplied by its own retention factor before the delta rule fires. This lets the model forget some dimensions of the state faster than others.

The Kimi K3 report calls this Equation (1). Every symbol:

• $x_t \in \mathbb{R}^d$ — the hidden state of token $t$.
• $q_t, k_t \in \mathbb{R}^{d_k}$ — query and key, both derived from $x_t$.
• $v_t \in \mathbb{R}^{d_v}$ — value, also from $x_t$.
• $S_t \in \mathbb{R}^{d_k \times d_v}$ — the recurrent state (small, fixed-size).
• $\alpha_t$ — per-channel retention (learned, per-token, per-head).
• $\beta_t$ — per-token write strength.
• $I - \beta_t k_t k_t^\top$ — an "erase" update along the current key (a full orthogonal projection only in the special unit-key, beta=1 limit).

Queries and keys use projection, ShortConv, Swish and L2 normalization. Values use projection, ShortConv and Swish without the same L2-normalization step. The report also applies head-wise RMSNorm and a full-rank learned output gate after the recurrence; the simple state equation is not the entire KDA layer.

4.4 Why Kimi K3 changed the log-decay parameterization

The tricky part in software is computing $S_t$ in parallel over a chunk of tokens, not one at a time. To do that, you need cumulative products of the decays, and those products can become astronomically small or large — overflowing in fp16/bf16.

Kimi Linear (the related hybrid model, not an alias for K2) mapped the raw decay logit $z$ through a negative-softplus: $g = -e^A \, \text{softplus}(z)$. This can go to $-\infty$, so the reciprocal $1/\gamma$ can overflow. Their fix was expensive: split each chunk into 16-token tiles and do the diagonal tiles with explicit position-pair loops that cannot use tensor cores.

Kimi K3 replaces this with a scaled sigmoid that bounds the log-decay from below:

The report attributes a faster intra-chunk path to this bounded-decay design. Its approximately 2.5× overall scaling-efficiency comparison includes architecture, training and data changes; it is not a measured 2.5× throughput gain caused solely by this kernel.

The original curve can be checked directly at A=0:

These are calculated function values, not hardware performance samples.

4.5 Chunkwise parallel form (skip if math-averse)

A chunked implementation must preserve the order of erase, decay and write. Rather than copying a compressed paper equation with ambiguous cumulative-decay indices, derive the recurrence directly. Define

Then $S_t=M_tS_{t-1}+U_t$. For a three-token chunk,

The order of matrix multiplication matters; decay and erase generally do not commute. A pair $(M,U)$ denotes the affine state map $S\mapsto MS+U$. Composition is associative:

Associativity explains how a scan can organize parallel work without making tokens noncausal. Materializing all these full transition matrices would be expensive; the report’s chunkwise formulation and specialized kernels exploit the diagonal-plus-low-rank structure instead. This derivation establishes the state algebra, not parity with its production kernel.

For the specified16-token tile, cumulative log-decay lies above−80 and the reciprocal is below $e^{80}$, within BF16 exponent range. It is not within FP16 range and is not a blanket bound for arbitrary chunk length or every intermediate operation. Underflow, rounding and accumulation precision still require separate analysis.

4.6 Gated MLA — the "occasional global attention"

KDA is fast but its state is a fixed-size matrix, which limits precision on very hard lookup tasks. So K3 interleaves one Gated MLA layer for every three KDA layers. MLA = Multi-head Latent Attention (from DeepSeek-V2), which is standard $\mathcal{O}(L^2)$ attention but with a compressed key/value cache. "Gated" adds a learned gate so the attention output can be attenuated when it isn't useful.

The intuition: KDA provides fixed-size recurrent memory; Gated MLA preserves a token-addressable global attention path. The reported3:1 repeating pattern is one evaluated tradeoff, not a universal optimum or a guarantee of exact retrieval.

5. Attention Residuals (AttnRes) — every layer talks to every earlier block

Story mode. In a normal transformer, information flows straight up the stack via residual connections: layer $\ell$'s input is layer $\ell{-}1$'s output. Earlier information can persist through the additive residual stream, but later layers normally see its accumulated mixture rather than selecting separately stored earlier block outputs. Attention Residuals let a later layer directly attend to the outputs of any earlier block — like a shortcut network across depth.

5.1 Standard residuals vs. attention residuals

A conventional residual stream carries an accumulated representation. Block AttnRes also retains separately selectable block-level sources at the same token position. The sources are sums of layer outputs, not arbitrary past-token keys.

The scoring path normalizes keys; the value path uses the unnormalized source vectors. Normalizing both would be a different operator. This diagram is depth selection, not sequence attention.

5.2 The AttnRes equations

A learned pseudo-query w is shared across token positions for one layer, but its scores depend on each position's stored representations. For available depth sources b_i, the full-form mechanism is

Normalize keys before scoring; values remain the stored source vectors. Omitting RMSNorm, as the earlier simplified equation did, changes the operator and lets source magnitude dominate the score. This is attention over depth at the same token position, not attention across sequence positions. The pseudo-query is learned, not generated independently from each token, while weights vary with token content.

Full AttnRes retains layer outputs, with memory proportional to depth. Block AttnRes sums layer outputs within each block and attends over the embedding, completed block sums, and the current block's partial sum where applicable. The pinned report says 8 blocks of nominal size 12 with a partial final block for 93 layers; counting the embedding yields 9 stored block sources. It is incorrect to infer either 32 layers from the 3:1 hybrid pattern or exactly 96 layers from 8 times 12.

Numeric example. Let b0=[1,0], b1=[0,2], and w=[1,0], ignoring epsilon/gain only for this hand calculation. RMS-normalized keys are [sqrt(2),0] and [0,sqrt(2)]. Scores are [sqrt(2),0], giving weights approximately [.8044,.1956]. The weighted raw values are [.8044,.3911]. If keys were not normalized, the scores and weights would differ. If values were also normalized, the output would differ again. Keep those three choices distinct.

The output is a learned mixture of earlier information, not a guarantee that early features or their gradients can never be lost. Additional states and communication create a tradeoff against a simple residual stream; ablations, not the diagram alone, support a quality comparison.

6. Stable LatentMoE — 16 experts out of 896

Story mode. A "Mixture of Experts" replaces a single big feed-forward network in each block with many small ones ("experts"), plus a router that picks which few to run for each token. The result: total parameters go up (more knowledge storage), but expert computation per token depends on the selected count and expert dimensions rather than the entire stored pool. Kimi K3 pushes this to an extreme — 896 routed experts, 16 activated per token, sparsity ratio 56. Expert count is an architectural quantity, not a measured diversity or quality score.

Extreme sparsity is hard for two reasons:

1. Activation explosion. Chained matmuls in the routed branch amplify outliers.
2. Load balancing. A large expert pool can make uneven routing and under-training harder to manage; neither failure is inevitable on every batch.

Stable LatentMoE introduces mechanisms intended to reduce both risks. Neither bounded activations nor a balancing rule eliminates every training or routing failure.

6.1 LatentMoE — cheap experts via a shared bottleneck

A conventional routed MoE can send the full $d$-dimensional state to each selected expert. Expert intermediate widths are architecture choices, not always $4d$. Selecting more experts increases logical expert work and dispatch volume at fixed expert shape, but actual network and weight traffic also depend on placement, batching, reuse and fused communication; a literal16× end-to-end cost is not implied.

LatentMoE (Elango et al., 2026) splits the model width into two pieces:

• A shared full-width path (2 shared experts in K3) that always runs.
• A compact latent path at width $\ell \ll d$: a shared projection $W^\downarrow$ takes the token down to $\ell$; each routed expert operates in $\mathbb{R}^\ell$; a single $W^\uparrow$ maps the aggregated result back to $\mathbb{R}^d$.

This is Equation (11) in the tech report. It is the payoff of the whole design: you can now afford 896 routed experts because they operate in a narrow latent space.

6.2 Normalized LatentMoE — one extra RMSNorm

Follow the routed branch past the selected experts. Their contributions are combined before normalization and the return to full width. Narrower experts reduce the cost of this branch, but they do not by themselves control the magnitude of that combined result.

Routing selection and physical expert placement are separate decisions. The diagram does not assign one GPU to each box or promise equal communication costs.

The routed aggregate $u$ can have wildly varying scale depending on which experts were picked and their weights. K3 inserts an RMSNorm before the up-projection. RMSNorm (Zhang & Sennrich, 2019) is just: divide the vector by the root-mean-square of its entries, then scale by a learned gain. The report’s ablation attributes improved validation behavior to this normalization in its evaluated setting. That is a publisher-reported result, not a guarantee for every MoE.

6.3 SiTU-GLU — a bounded activation

A SwiGLU FFN uses $\operatorname{Swish}(W_gx)\odot(W_ux)$, followed by an output projection. The gate input is the projected vector $W_gx$, not the unprojected $x$. Both factors are unbounded, so if a "gate" and "up" projection happen to spike together on one channel, you get huge activations — which overflow in low-precision arithmetic (a real problem for MXFP4/FP8 training).

K3 introduces SiTU-GLU (Sigmoid Tanh Unit GLU), which "smoothly caps" both branches with a scaled tanh:

with $\beta_1 = 4$ and $\beta_2 = 25$. Two nice properties:

• Near the origin, $\beta \tanh(z/\beta) \approx z$ — so the bounded branches approximate their unbounded counterparts there. Whether most actual inputs are in that region is an empirical distribution question.
• Far from the origin, both branches saturate. The elementwise product before the FFN output projection is bounded: $\|\text{SiTU-GLU}(x)\|_\infty \le \beta_1 \beta_2 = 100$. A subsequent learned output matrix can amplify it; this is not a bound on the entire layer or residual stream.

For the same scalar input in both projected branches, compare the actual products, not a one-branch gate sketch:

The bound is on this product before a learned output projection; no universal overflow immunity follows.

6.4 Quantile Balancing — next-step bias from margin quantiles

The problem. In MoE, some experts get overwhelmed with tokens ("hot"), others starve ("dying"). With 896 experts, this is a serious production risk: hot experts slow down training, dying experts waste parameters.

Prior state of the art (DeepSeek-V3) added a per-expert bias $b_j$ to the router score, and nudged $b_j$ by $\pm \gamma$ at every step depending on whether expert $j$ was over- or under-utilized. Simple. But at 896 experts the fixed step size $\gamma$ becomes very hard to tune: too small → slow to adapt, too large → oscillates.

K3's idea. Estimate an expert-specific bias using margins relative to each token's current Top-(k+1) cutoff. The actual routes take the first k entries of the biased scores. For m tokens, n experts and k selected experts, the target load is q=mk/n. Holding cutoffs fixed and assuming no ties, choose the threshold at the (q+1)-th largest margin so q margins exceed it. This motivates the quantile update, not a guarantee that a new batch and recomputed coupled top-k choices will have perfectly equal counts.

Here s is the raw sigmoid router score; the old bias enters through the cutoff alpha. Selection uses s+b, but mixture weights renormalize the selected raw s values, excluding the bias. The common bias offset does not alter top-k selection. To preserve causal batch processing, the update takes effect on the next step. Biases are frozen at inference.

The report uses histograms with a few hundred bins per expert and one all-reduce of counts, not the earlier claimed 8000 buckets. Approximation resolution, ties and changes in the token distribution are relevant limits. Perfectly balanced expert execution in the separate MoonEP systems layer is not the same claim as exact router load equality for every future inference batch.

6.5 Per-Head Muon — one optimizer detail worth mentioning

Muon is a recent optimizer for hidden-layer weights (Jordan et al., 2024, scaled by Moonshot in 2025) that updates whole weight matrices with a specially-normalized step. K3 extends this to treat each attention head as its own optimization group, which the report motivates by unequal gradient/momentum scales across heads. Its experiments report more balanced updates; different head widths or specialization rates are not required by that argument.

7. Vision — MoonViT-V2

The pinned report describes joint language/vision training from the start rather than post-hoc alignment. Images and video frames go through MoonViT-V2, a vision transformer trained jointly with the language model. A small MLP projector maps its visual features into the same embedding space the text tokens live in.

Practical consequence: the model can, mid-reasoning, execute code to zoom or crop an image, feed the result back as a new visual observation, and continue — the "vision-in-the-loop" pattern the blog highlights for game-dev and CAD tasks.

The report specifies a27-layer, approximately401M-parameter MoonViT-V2 with14-pixel patches and12 attention heads. It uses RMSNorm and bias-free linear/attention projections. Shared image/video weights support factorized within-frame spatial and between-frame temporal attention; temporal pooling compresses time. A2×2 pixel-shuffle reduction cuts visual token count by four before the MLP projector. It is not a claim that arbitrarily long high-resolution video fits at zero cost. The paper compares from-scratch next-token-trained vision initialization with a SigLIP-initialized baseline; this ablation does not prove contrastive pretraining is useless for every multimodal system.

8. Training — what actually happened

8.1 Pre-training

• Scale. 2.8T parameters, 104B active, total/activated parameter ratio ≈27; routed-expert selection sparsity is896/16=56. These are different ratios.
• Data. Refined recipes over Kimi K2; the report emphasizes better data curation contributed to the 2.5× win alongside architecture.
• Efficiency. Kernel-level co-design: fused kernels for KDA, sequence parallelism for AttnRes, custom kernels for the latent MoE.

The reported text corpus has Web, Code, Mathematics and Knowledge domains with rule/classifier filters, deduplication and smaller-model mixture ablations. Vision includes captions, interleaved text/images, OCR, perception, video and code/rendered-visual pairs. The training recipe uses cosine decay with1% warmup and weight decay.1;8K→64K pretraining is followed by256K→1M cooldown. NoPE avoids retuning explicit RoPE parameters, but the report still trains on long natural/synthetic examples. A1M capacity is not evidence of perfect long-range retrieval. The2.5× comparison comes from fitted OOD-validation-loss versus FLOP curves for the combined model/data/training family, not a measured kernel-throughput ratio.

8.2 Post-training pipeline

Pre-training gives the model a broad prediction objective; post-training must turn that capability into useful behavior under different task and token budgets. The reported stages are supervised fine-tuning for a cold-start policy, domain/effort-specific reinforcement learning, and multi-teacher on-policy distillation into one policy. Read them as demonstration, specialization, and consolidation. The last stage matters because the deployed model must bring the specialist policies back together.

8.3 Reinforcement Learning across "effort levels"

The middle box in that pipeline is not a single uniform workload. A short answer and a long coding task need different spending limits even when both are judged on success.

The team trains the model at multiple reasoning-effort levels ("low", "high", "max") within each domain, then merges them. This is what lets the same K3 model spend a small budget on an easy question or a much larger budget on a difficult agentic task — while explicitly evaluating the quality/cost tradeoff. Consolidation can still regress a particular task or effort level; the architecture does not rule that out.

8.4 On-policy distillation with dense rewards

Instead of the usual "sparse verifier" reward, K3 uses a dense token-level KL-style reward comparing the student to a domain-specialist teacher:

Every symbol: $e$ = effort level, $d$ = domain, $\text{sg}$ = stop gradient, $R_{\max}$ = clipping threshold to prevent extreme advantages. Every generated token gets a reward — much denser signal than "success/failure at the end of a trajectory," which makes long-horizon, tool-using trajectories easier to train.

8.5 Unified white-box RL environment

Every "agent harness" (Kimi Code, Claude Code, Codex, OpenClaw, Hermes) is decomposed into interchangeable modules: tools, prompts, context management, memory, subagents. During RL, tasks are matched with randomized harness configurations. This is intended to reduce dependence on one harness. It does not prove transfer to an unseen harness: tool schemas, permissions, context management and failure handling still need matched evaluation.

8.6 Knowledge-graph-guided task synthesis

Post-training tasks are only as good as their source material. Moonshot builds a self-evolving, hierarchical knowledge graph (agents recursively explore and add nodes), then samples related nodes as keyword sets to guide web retrieval and task synthesis.

8.7 Why nine teachers exist

K3 does not train one specialist for every benchmark. It defines three broad RL domains:

1. General tasks — knowledge, reasoning, vision, faithfulness, search, and professional work.
2. General agents — long-horizon assistance, deep research, and extended writing.
3. Coding agents — software engineering, kernels, coding experience, and web development.

Each domain is trained at three reasoning-effort levels: low, high, and max. The Cartesian product gives $3\times3=9$ expert policies. Multi-Teacher On-Policy Distillation then consolidates them into one deployed policy conditioned on the requested effort.

This separation addresses a real optimization conflict. A policy rewarded only for maximum task success tends to spend too many tokens on easy problems. A policy rewarded only for brevity gives up too early on hard problems. K3 trains the effort points explicitly rather than hoping one reward weight produces all three behaviors.

8.8 Per-problem reasoning budgets

Let $b_0(x)$ be an initial token budget estimated for problem $x$, and let $T(y)$ be the token cost of trajectory $y$. K3 applies a budget multiplier $\tau$ and overrides the task reward with $-1$ when

For a general reasoning task, $T(y)$ counts thinking tokens. For an agentic task it counts cumulative output, including reasoning traces and tool-call arguments. Training begins with a larger $\tau$ for the max-effort policy, then anneals $\tau$ to derive high- and low-effort experts.

The important idea is relative budgeting. A fixed 8,000-token cap treats a one-line arithmetic problem and a repository-wide refactor identically. Scaling the budget from a problem-specific baseline creates a more meaningful pressure toward efficiency.

8.9 Partial rollouts and the straggler problem

Suppose training samples $K$ completions for each of $N$ prompts, so $NK$ trajectories run concurrently. Tool-using trajectories have heavy-tailed duration: one may finish in seconds while another waits on a build, browser, or sandbox for minutes. A synchronous trainer that waits for the slowest trajectory leaves expensive accelerators idle.

K3 pauses generation after a fraction $\lambda\in(0,1)$ of trajectories complete:

Optimization can start on completed prompt groups, not on every isolated finished trajectory. Paused trajectories are queued and prioritized for resumption in the next iteration. A prompt group becomes ready only once all $K$ responses for that prompt finish; reaching the global completion fraction alone does not imply that any particular group is ready.

The cost is policy staleness: one long trajectory can span several optimizer updates, so stored tokens may have been sampled by policies older than the policy at optimization time, and one trajectory can contain multiple policy versions. K3’s optimizer uses per-token regularization to keep updates in a localized neighborhood, making the extreme off-policy mixture tolerable.

8.10 Reward models, rubrics, and verbosity hacking

Many coding and mathematics tasks have executable verifiers. Open-ended research and professional writing do not. K3 uses an Agentic Generative Reward Model for these tasks. The judge must follow a protocol:

1. Read the outcome or artifact.
2. Generate a task-specific rubric.
3. Score each candidate against that rubric.
4. Record the rubric scores in a scorepad.

The protocol makes the basis of comparison explicit. It does not eliminate judge error, but it is more auditable than an unexplained scalar.

Longer answers can appear more thorough and exploit a judge’s preferences. To reduce that reward hack, K3 estimates an initial verbosity $\ell_0$, chooses multiplier $\sigma$, and automatically loses a comparison when output length exceeds $\sigma\ell_0$. This is the output-length analogue of reasoning-budget control.

8.11 Multi-Teacher On-Policy Distillation, line by line

For domain $d$, effort $e$, query $x$, prefix $y_{<t}$, sampled token $y_t$, teacher $\pi_{\text{teacher}}^{(d,e)}$, and student $\pi_\theta$, K3 defines

Interpretation:

• If the teacher assigns the sampled token more probability than the student, the log-ratio is positive.
• If the student already overweights it, the ratio is negative.
• sg means stop gradient: treat the reward number as fixed rather than differentiating through its construction.
• Clipping to $[-R_{\max},R_{\max}]$ prevents rare tiny probabilities from producing enormous, destabilizing advantages.
• Because this reward exists at every generated token, it gives denser credit than one terminal pass/fail signal.

The report also states a useful negative result: more fine-grained top-$k$ distillation objectives did not clearly improve convergence or final performance in their setting. Reporting failed refinements matters; it prevents the architecture story from becoming survivor-biased mythology.

8.12 Deployment-aware learning starts before deployment

K3 applies quantization-aware training through SFT and RL, not only after alignment. Routed expert weights use MXFP4 and their input activations use MXFP8, while attention projections, latent MoE projections, routers, and shared experts remain at higher precision.

That alignment between rollout precision and serving precision removes a common mismatch. If RL explores with high-precision weights but production serves a separately quantized approximation, the deployed policy can encounter action probabilities it never optimized under. K3 makes quantization noise part of the policy’s training environment.

9. Infrastructure — the publisher’s reported design

The tech report spends the most pages here, because at 2.8T parameters, infrastructure is the model.

9.1 MoonEP — perfectly balanced expert parallelism

Expert parallelism means: different GPUs hold different experts. If token routing is imbalanced, some GPUs sit idle. MoonEP enforces static computation shapes (each GPU processes the same fixed number of tokens per step) with zero-copy communication. Balanced token counts under that planning scheme are distinct from equal wall-clock time: network contention, expert placement, kernel behavior and scheduling can still cause stalls. The mechanism is not a proof of universal latency predictability. MoonEP dynamically plans and migrates redundant experts, stages their backward gradients locally and reduces them to home ranks. The report proves a feasibility bound of E/R redundant-expert slots per rank under its formal setup; its GPU planner is distinguished from exact offline ILP reference solutions. Balance of scheduled expert work is not identical to equal wall-clock duration.

9.2 KDA Context Parallelism

For the report’s model and training workload, million-token context motivates sequence sharding; feasibility depends on tensor sizes, precision, activations, caches and device capacity, not token count alone. Context parallelism shards the sequence across GPUs and communicates the KDA recurrent state at chunk boundaries. K3 pairs this with state-aware prefix caching so that if two requests share a long prefix (very common in coding sessions), the KDA state at the shared boundary can be restored from cache instead of recomputed.

9.3 KDA-aware prefix cache — the deployment story

Normal LLM serving caches the "KV cache" of long prefixes so a second request with the same prefix skips prefill. For a 1M-token coding session with a 400K-token prefix, this is life-or-death for latency.

K3 has two caches per block: the MLA KV cache (grows per token) and the KDA recurrent state (fixed size, one per request). A cached prefix is only reusable if both are restorable at the same boundary. K3’s design:

The report's Figure12 uses a6,144-token physical block containing twelve512-token hash blocks. A2,800-token matching prefix can reuse2,560 tokens only when all required recurrent groups have checkpoints at that endpoint. Hashing and allocation granularity are distinct. The longest MLA match alone is insufficient. Pin all hit blocks before private allocation, copy before mutation, and keep state-export boundaries stable across scheduling interleavings. These are reported engine contracts, not a service exercised here.

Operational consequence: prefix reuse can avoid repeated work only when both the MLA token cache and every required KDA state checkpoint are present at compatible boundaries. A cached state must be copied into private mutable request state before advancing it. Missing one KDA group invalidates the compound hit. This is a consistency and isolation contract, not merely a hit-rate statistic. API prices and future workload cache-hit rates are mutable and are not asserted here.

9.4 AgentENV — resumable microVM sandboxes for agentic RL

RL for a coding agent needs an environment where the agent can safely run arbitrary code. Containers and microVMs offer different isolation and startup tradeoffs; neither is categorically secure without its threat model and configuration. AgentENV uses Firecracker microVMs with incremental checkpointing:

• Checkpoint latency: as low as 133 ms in the report
• Resume latency: as low as 49 ms in the report
• Pause/Resume: the report says paused guest state consumes no running CPU/memory; persisted snapshots and the supporting storage/control plane are not free
• Fork: isolate changes to a copied guest state for judging; external network/storage effects still require their own isolation policy
• Snapshot: for recovery

During K3 training and evaluation, 51.2 million sandboxes were created across 1.5 million distinct images. AgentENV is open-sourced here.

9.5 Fleet-level scheduling

Two policies keep production serving predictable:

• Cache-aware affinity scheduling. Consistent hashing assigns a primary cache-holding cluster and a secondary. The report explicitly says the secondary does not hold that session’s prefix cache and must re-prefill on failover; it is not a hot-cache standby. Spreading secondary assignments distributes the re-prefill work.
• Budget-based admission control. Separate budgets for short and million-token request classes limit cross-class interference. This is the reported admission-control design, not a guarantee against every overload, shared dependency or SLO failure.

10. Deployment — how it stays cheap

10.1 MXFP4 quantization-aware training

MXFP4 is a 4-bit floating point format (Rouhani et al., 2023). K3 quantizes MoE expert weights (which dominate memory) to MXFP4, with activations at MXFP8, keeping attention and shared experts in higher precision. Quantization-Aware Training (QAT) is used throughout SFT and RL — so the model is trained with the exact quantization scheme it will be served under. This targets a major source of train/serve numerical mismatch; it does not prove all runtime kernels and numerical paths are identical.

10.2 EAGLE-3 speculative decoding

K3's multi-token-prediction (MTP) head is fine-tuned into an EAGLE-3 draft model: a smaller network unrolled for seven steps during the report’s draft-training procedure. The target verifies proposed continuations; seven training steps do not establish a fixed deployment draft length for every request. Instead of the standard KL loss, the draft is trained directly on the LK loss:

which is the negative log of the distribution overlap. For normalized target and draft distributions, the overlap equals the expected single-step acceptance probability under the specified lossless speculative-sampling rule. End-to-end throughput also depends on draft cost, verification batching and accepted-prefix length; the loss alone is not a measured speedup.

10.3 KDA state replay for speculative decoding

A subtle problem: if a draft window includes tokens that are rejected, the KDA state has already advanced beyond the accepted prefix. Rolling back a large state per step is expensive. K3 caches only the projected inputs of drafted tokens (much smaller than the state), then replays the accepted prefix on-chip in a single fused kernel. This is the concurrent-work "ReplaySSM" idea, integrated cleanly into the serving stack.

11. Benchmarks — what they mean

The mechanisms above explain what the system is designed to do; benchmarks ask how it performed in a particular evaluation. Read each benchmark separately, with its harness and reasoning budget from the blog’s footnotes. The original chart mixed GDPval Elo with other score scales; compare within a row using a named protocol, not by averaging across benchmarks.

How to read each benchmark:

• DeepSWE — software-engineering issues from real GitHub repos; the model is scored on whether its patch passes tests.
• Terminal-Bench 2.1 — realistic command-line tasks; the harness runs commands and grades outcomes.
• FrontierSWE / SWE-Marathon / KCB 2.0 / ProgramBench — variations on "solve real bugs / write real code."
• BrowseComp — an agent has to browse the web and produce a verifiable answer. K3 is highest among the compared systems in the pinned table; this is not an all-time universal ranking.
• AutomationBench / JobBench / AA-Briefcase — agentic work benchmarks with distinct tasks and grading protocols. Human-authored tasks do not imply that every result is human-graded. The pinned table reports AA-Briefcase as Elo, unlike the percentage-like AutomationBench and JobBench scores.
• CharXiv / ZeroBench / MMMU-Pro / PerceptionBench — vision-heavy reasoning.

The reported pattern is strongest on several long-horizon agentic tasks (BrowseComp, AutomationBench, SWE-Marathon), which motivates looking closely at context handling, caching, and RL environments. These are plausible connections, not a component-level causal result: isolating any one contribution requires the ablations, not just the leaderboard.

The pinned report (§6.1.3) and README specify maximum reasoning effort for Kimi K3, temperature 1.0, top-p 0.95 for single-step tasks and 1.0 for agentic tasks. The saved publisher blog footnote instead says top-p 1.0 for all listed results. This is a source discrepancy, not a license to silently combine protocols; the table below follows the pinned README, and a reproduction should record the actual per-task configuration. They also describe different harnesses across compared systems. For example, the repository reports DeepSWE 67.5 with the Kimi Code harness while noting 67.3 with mini-SWE-agent. This is evidence that a score belongs to a model-plus-protocol, not a context-free model essence.

The official table includes percentage-like task scores and Elo ratings. Do not average BrowseComp, pass@5 and an Elo value into a single “overall intelligence” score. They have different definitions, ranges and uncertainty. Publisher comparisons are useful leads for a controlled application evaluation, not proof that a specific architecture component caused a reported advantage.

For an application, define relevant constraints first: authorized evidence use, answer support, abstention, latency, cost and operating environment. A much smaller model or a non-generative evidence view may meet those constraints. A larger parameter count is not a requirement created by the product.

Dated comparison table: keep each metric separate

Selected official README rows at the pinned revision; values are reported results, not rerun here. Column labels preserve publisher model identities. Fable5 may include fallbacks and GPT5.6Sol potential cyberguards in the reported setup. A dash is unreported, not zero. CharXiv/MMMU pairs are no-tool/tool values; ZeroBench is pass@5, not single-attempt accuracy. GDPval is Elo.

For example, DeepSWE67.5versus67.3with a different harness is not an isolated model gain. BrowseComp91.2uses the stated compaction protocol; the publisher reports90.4with1Mcontext and no context management. Read source/task versions and missing cells before interpreting a ranking. The earlier invented “MMMU-Pro-ish=0” row is replaced with the actual named benchmark.

12. Case studies (publisher demonstrations, not reproduced here)

12.1 MiniTriton — a reported compiler-building case study

The publisher blog describes MiniTriton as a DSL frontend, tile-level IR over MLIR, optimization passes and PTX code generation. It reports supported roofline workloads comparable to or faster than Triton/torch.compile and nanoGPT loss curves closely tracking a reference with minor divergence. That is narrower than exact FP32 parity or universal compiler superiority. The project is a case study to audit: identify the exact generated revision, dependencies, accepted tests, baseline kernels and geometric-mean benchmark denominator. A successful matmul example is not proof of a complete compiler or numerical parity over all dtypes. No compiler build or GPU kernel was run here.

12.2 nano-KPU — simulation versus manufactured silicon

The publisher reports a48-hour nano-KPU run using open EDA tools and Nangate45, with a4mm² design closing timing at100MHz and over8,700 tokens/s in simulation. The report describes a nano architecture with hybrid KDA/NoPE-MLA, Block AttnRes and group-wise INT4 MoE, not a physical2.8T-model accelerator demonstration. Separate RTL simulation, synthesis estimates, timing closure under a particular standard-cell library and manufactured silicon. A simulated tokens-per-second number is not measured physical-device throughput. Review testbenches and assumptions before attributing general chip-design capability.

12.3 I–Love–Q astrophysics reproduction

For I–Love–Q, the publisher reports relating neutron-star moment of inertia, tidal Love number and quadrupole moment, reading20+ papers and evaluating300+ equations of state in about two hours. Its weeks-of-human-work comparator is an unverified estimate, not a controlled productivity experiment. A scientific reproduction needs source equations, unit conventions, boundary conditions, convergence studies and comparison data. A dashboard or plausible curve is not verification. This audit does not certify the astrophysics calculation.

12.4 GWTC-5 gravitational-wave analysis

The blog describes a GWTC-5 case study with391 events,20+ concurrent subagents,7 figures,2 tables and10+ papers. These are attributed artifact counts, not verification of the astrophysical conclusions or an independently authenticated catalogue release. Check dataset release, selection criteria, uncertainty propagation and independence of parallel subtasks. Multiple agents can repeat the same mistaken assumption; concurrency does not increase evidential independence by itself.

12.5 Video editing from 56 clips

The publisher describes editing a teaser from56 source clips, including selection, motion-matched cuts, beat synchronization, audio and repeated revision. Treat clip selection, temporal alignment, audio synchronization and rendering as separate acceptance checks. Native vision can supply observations, but an editing pipeline still uses tools and multiple computation stages; understanding and every editing action do not happen in one forward pass.

13. Limitations and compatibility

Context capacity is not demonstrated accuracy on every million-token task. Fixed-size recurrent memory can mix or lose associations; the remaining MLA cache still grows with context. Benchmarks depend on effort, harness, sampling and task versions. Reported systems measurements are workload-specific, not a capacity guarantee for a different fleet.

The pinned report’s AppendixF distinguishes thinking and instruct modes. Thinking mode retains the think channel in history, even when empty; instruct mode retains response and tools channels. Preserve the documented message/template contract when serving this model. Dropping fields under a generic chat template can create a distribution mismatch; that is not a claim that every OpenAI-compatible API necessarily drops them or cannot support the model.

14. What this means for open-weight engineering

The publisher presents the release as model artifacts plus inspectable descriptions of recurrent sequence mixing, depth routing, sparse channel computation and supporting systems. Open weights are not by themselves a license audit or a fully reproducible pretraining pipeline. Public repositories for selected components do not establish that every dependency, dataset, production control plane or training run is reproducible.

Compare the architectural mechanisms through controlled ablations and the application through a matched evaluation. A benchmark gap does not prove either an architectural gap or the absence of one. Claims of being the first model above a size threshold need a defined comparison set/date; this lesson does not make that historical priority claim.

The publisher also reports sensitivity to missing thinking history and a tendency to make unexpected decisions under ambiguous requests. These are behavioural risks to test with the documented template and explicit tool permissions, not reasons to grant broader autonomy. Preserve model-specific channels according to AppendixF, but enforce authorized actions in the controller rather than relying on an AGENTS.md instruction alone. Its named proprietary comparisons are a dated publisher evaluation, not a current universal UX ranking.

15. Glossary (skim this after)

• Recurrent state: a fixed-shape matrix updated as tokens arrive; not a lossless list of every past token.
• MLA cache: compressed per-token attention information; its token dimension still grows.
• Depth source: an embedding, completed block sum or current partial sum used by AttnRes at a token position.
• Latent expert path: shared down-projection, selected expert transformations, normalized aggregation and up-projection.
• Routing bias: a selection-control term; it is excluded from the normalized raw-score mixture weights in the reported Quantile Balancing rule.
• QAT: training with a quantization-aware numerical path; not proof all serving implementations behave identically.
• On-policy distillation: student-generated trajectories scored with a teacher distribution; not simply replaying teacher-written answers.
• Acceptance probability: probability a draft token is retained under a specified target/draft sampling rule; not the same as tokens-per-second speedup.

16. Optional prerequisite video path

These optional external pointers are not primary evidence for Kimi-specific claims. Video availability and metadata may change; the examples here do not verify the videos’ instructional completeness. Use the pinned report for model-specific facts and the written checkpoints below to identify prerequisite gaps. The four-week schedule is an adjustable example, not an evidence-backed mastery deadline.

Tier 0 — first understand AI and language models

AI is the broad field of building systems for tasks associated with intelligence. Machine learning fits behaviour from data; deep learning uses multilayer learned representations. Generative modelling specifies how to produce samples or sequences, whereas a language model models language/token structure. An agent is a controller that uses a policy with observations, state and authorized actions; it can wrap a language model but is not identical to one forward pass. These are overlapping categories, not competing brand names.

1. Large Language Models explained briefly — 3Blue1Brown — the quickest visual orientation.
2. AI, Machine Learning, Deep Learning and Generative AI Explained — IBM Technology — separates terms that are often used interchangeably.
3. Intro to Large Language Models (1-hour talk) — Andrej Karpathy — the best single broad introduction.
4. How I use LLMs — Andrej Karpathy — connects the architecture to practical use.

Checkpoint: You should be able to explain the difference between AI, machine learning, deep learning, generative AI, a language model, and an agent.

Tier 1 — vectors, matrices, probability, and loss

1. Vectors — Essence of linear algebra, Chapter 1 — 3Blue1Brown
2. Linear transformations and matrices — Chapter 3 — 3Blue1Brown
3. Matrix multiplication as composition — Chapter 4 — 3Blue1Brown
4. Eigenvectors and eigenvalues — Chapter 14 — 3Blue1Brown — optional, but useful for low-rank and conditioning discussions.
5. Neural Networks Part 5: ArgMax and SoftMax — StatQuest
6. Neural Networks Part 6: Cross Entropy — StatQuest

Checkpoint: Given $y=Wx$, identify the shapes of $x$, $W$, and $y$; explain dot product versus outer product; turn logits into probabilities; and explain why $-\log p_y$ punishes confident mistakes.

Tier 2 — how neural networks learn

1. But what is a neural network? — 3Blue1Brown
2. Gradient descent, how neural networks learn — 3Blue1Brown
3. Backpropagation, intuitively — 3Blue1Brown
4. Backpropagation calculus — 3Blue1Brown
5. Gradient Descent, Step-by-Step — StatQuest — slower alternative.
6. Neural Networks Part 2: Backpropagation Main Ideas — StatQuest — slower alternative.
7. MIT Introduction to Deep Learning 2025 — 6.S191 — university-level consolidation.

Checkpoint: Explain a forward pass, loss, gradient, chain rule, backward pass, learning rate, optimizer step, and why exploding activations destabilize training.

Tier 3 — Transformers and attention

1. Transformers, the tech behind LLMs — 3Blue1Brown
2. Attention in transformers, step-by-step — 3Blue1Brown
3. How might LLMs store facts — 3Blue1Brown — especially useful before MoE because it explains the FFN block experts replace.
4. Transformer Neural Networks, ChatGPT’s foundation, Clearly Explained — StatQuest
5. What are Transformers? — IBM Technology
6. AI Language Models & Transformers — Computerphile
7. Let’s build GPT from scratch — Andrej Karpathy — code-level capstone.
8. Stanford CS336 Lecture 1: Overview and Tokenization — advanced course entry point.

Checkpoint: Derive scaled dot-product attention, state the shapes of Q/K/V, explain causal masking, identify attention versus FFN computation, and explain why full attention is quadratic in sequence length.

Tier 4 — MoE, MLA, long context, RL, and efficient inference

1. Mixture of Experts: How LLMs get bigger without getting slower — Julia Turc
2. How DeepSeek Rewrote the Transformer — MLA — Welch Labs
3. How KV Cache Works — Toni
4. Reinforcement Learning from Human Feedback: From Zero to ChatGPT — Hugging Face
5. RLHF explained with math derivations and PyTorch code — Umar Jamil
6. State of GPT — Andrej Karpathy
7. Deep Dive into LLMs like ChatGPT — Andrej Karpathy
8. Speculative Decoding: How a Draft Model Makes LLMs Faster — Devsplainers
9. LLM Quantization Explained — Joydeep Bhattacharjee
10. Mamba and S4 Explained: Architecture, Parallel Scan, Kernel Fusion — Umar Jamil — useful comparison for recurrent long-context designs.

Checkpoint: Explain top-k expert routing, load imbalance, KV-cache growth, latent KV compression, policy/rollout/reward, quantization-aware training, and draft-then-verify decoding.

Tier 5 — vision and multimodal models

1. An Image is Worth 16×16 Words: ViT paper explained — Yannic Kilcher
2. Coding a Multimodal Vision-Language Model from scratch in PyTorch — Umar Jamil
3. CS231n Lecture 10: RNNs, Image Captioning, LSTM — Andrej Karpathy / Stanford — optional historical context.

Checkpoint: Explain image patches, a vision encoder, projector, visual tokens, spatial versus temporal attention, and why native multimodal pre-training differs from attaching a frozen encoder later.

A practical four-week schedule

• Week 1: Tier 0 + Tier 1. Rework the shape trace in §2.13 by hand.
• Week 2: Tier 2 + the first six videos in Tier 3. Build one tiny neural network or follow micrograd.
• Week 3: Finish Tier 3, then read §§3–6 of this post while watching Tier 4 videos 1–3.
• Week 4: Tier 4 videos 4–10 + Tier 5. Read the K3 technical report’s architecture, post-training, and infrastructure sections beside this post.

Do not try to memorize every acronym. At the end of each tier, explain the checkpoint aloud without notes. If you cannot, rewatch only the segment tied to the missing concept.

17. Reading path if you want to go deeper

1. The blog post first. kimi.com/blog/kimi-k3 has the images and case-study videos we couldn't inline here.
2. The tech report, Section 2 (architecture) — with this post open in a second tab, every equation should now be readable.
3. Kimi Linear paper (arXiv 2510.26692) — the original KDA introduction.
4. DeepSeek-V3 tech report (arXiv 2412.19437) — the auxiliary-loss-free routing K3 improves on, and the MLA baseline.
5. LatentMoE (arXiv 2601.18089) — the paper Stable LatentMoE builds on.
6. Muon optimizer (Jordan et al., 2024) — for Per-Head Muon.
7. EAGLE-3 (arXiv 2503.01840) — for the draft-model design.

Worked KDA state update and limits

Let the previous state be S=[[1,2],[3,4]], alpha=[.5,.25], normalized key k=[1,0], value v=[2,-1], beta=.5, and query q=[1,0]. First decay rows: D S=[[.5,1],[.75,1]]. The old value at k after decay is [ .5,1 ]. The desired correction is v minus this value = [1.5,-2]. Write beta*k times that correction, adding [[.75,-1],[0,0]] to obtain the new state [[1.25,0],[.75,1]]. Reading S-transpose q gives [1.25,0]. This is the same result as applying the erase matrix then the write outer product.

The order matters: row-wise decay before erase is not generally interchangeable with erase before decay when k has multiple nonzero coordinates. A normalized key and beta in (0,1) describe a partial overwrite, not deletion of all old information. Other key directions can interfere in the same fixed-capacity matrix. Retaining this matrix across a million tokens does not prove exact recovery of every token. The hybrid MLA layers maintain a different access path and a cache growing with sequence length.

import numpy as np
S = np.array([[1., 2.], [3., 4.]])
alpha = np.array([.5, .25])
k = np.array([1., 0.]); v = np.array([2., -1.]); beta = .5
D = alpha[:, None] * S
new = (np.eye(2) - beta*np.outer(k, k)) @ D + beta*np.outer(k, v)
delta_form = D + beta*np.outer(k, v - k@D)
np.testing.assert_allclose(new, [[1.25, 0.], [.75, 1.]])
np.testing.assert_allclose(new, delta_form)
np.testing.assert_allclose(new.T @ k, [1.25, 0.])
assert np.exp(80.) < np.finfo(np.float32).max
assert np.exp(80.) > np.finfo(np.float16).max
print('KDA update, delta-form equivalence and exponent-range distinction pass')

This CPU test checks one recurrence and a float32-exponent proxy for the BF16 range; it is not an implementation or benchmark of the full KDA layer, chunkwise UT transform, BF16 hardware or FlashKDA kernels.

Practice: keep the three axes separate

1. Sequence: show which storage is constant in token count for KDA and which grows for MLA. Answer: the recurrent state per head is fixed-size; MLA retains token-dependent latent cache entries. Total hybrid memory is not constant.
2. Depth: why can a pseudo-query be fixed while attention weights depend on the token? Answer: the depth-source keys vary at each token position.
3. Width: compare 16/896 to 104/2800. Answer: approximately 1.79% is the routed-expert selection fraction; approximately 3.71% is the reported activated/total parameter ratio. Dense/shared paths explain why they need not match.
4. Numerics: why does bounding log-decay at -5 not make a 16-token reciprocal FP16-safe? Answer: exp(80) far exceeds FP16's maximum, though it is below the BF16 exponent maximum.
5. Routing: can the quantile rule guarantee equal load on the next batch? Answer: not generally; fixed-cutoff/no-tie derivation and histogram estimates are not a promise for shifted future data.
6. Evidence: does a high agent benchmark identify the cause as KDA? Answer: no; architecture, post-training, tools, context policy and harness all participate. Isolated ablations address narrower causal questions.

Capacity and evaluation checks

1. Why is multiplying 2.8T by 16/896 not a valid reconstruction of the official 104B active count?
2. Is the idealized 1.4 TB payload an allocation plan for a running inference service?
3. Two systems score 67.5 and 67.3 under different harnesses. Have you established a model-only improvement?

CPU laboratory: recurrence, depth, experts and replay

The next program is a complete generated-fixture laboratory, not Kimi inference. It connects the three axes in one reproducible artifact: recurrent state across tokens, normalized-key/raw-value selection across depth, and a small latent expert branch. It then tests the control mathematics used in post-training and speculative serving. Dimensions are deliberately tiny, all weights are generated locally, and no model weights, datasets, GPU or network are needed.

The scalar KDA oracle uses explicit index loops instead of the vector implementation. A wrong erase/decay order must disagree. An affine scan checks associativity without claiming the production UT/Triton kernel has been reproduced. In the depth example, normalizing values as well as keys must change the output. In routing, adding a common bias cannot change the chosen experts, but a frozen-cutoff quantile update need not perfectly balance a new batch. Repeated Top-(k+1) cutoff margins can tie even when the raw score matrix has distinct entries; strict thresholding can then select fewer than the target count.

For the illustrative decoder state, a prefix cache must contain both the recurrent state and the same-boundary token cache. The toy token cache stores values rather than actual MLA latent projections; it tests the boundary/copy/replay contract only. Rejected draft tokens must not remain in either cache. Production additionally requires model/template/precision identity, authorization, position/convolution state and all layer-group checkpoints.

import numpy as np
 
rng = np.random.default_rng(803)
 
def unit(x):
    return x/np.linalg.norm(x, axis=-1, keepdims=True)
 
def kda_step(S, k, v, alpha, beta):
    decayed = alpha[:, None]*S
    return decayed + beta*np.outer(k, v-k@decayed)
 
def scalar_step(S, k, v, alpha, beta):
    out = np.zeros_like(S)
    for i in range(S.shape[0]):
        for j in range(S.shape[1]):
            old_read = sum(k[r]*alpha[r]*S[r, j] for r in range(len(k)))
            out[i, j] = alpha[i]*S[i, j]+beta*k[i]*(v[j]-old_read)
    return out
 
L, dk, dv = 7, 3, 2
K, Q = unit(rng.normal(size=(L, dk))), unit(rng.normal(size=(L, dk)))
V = rng.normal(size=(L, dv))
A = rng.uniform(.2, .95, (L, dk))
B = rng.uniform(.1, .9, L)
S0 = rng.normal(size=(dk, dv))
S, scalar = S0.copy(), S0.copy()
outputs, transitions = [], []
for k, q, v, a, b in zip(K, Q, V, A, B):
    S = kda_step(S, k, v, a, b)
    scalar = scalar_step(scalar, k, v, a, b)
    np.testing.assert_allclose(S, scalar, atol=1e-12)
    outputs.append(q@S)
    transitions.append(((np.eye(dk)-b*np.outer(k, k))@np.diag(a), b*np.outer(k, v)))
 
def compose(later, earlier):
    M2, U2 = later
    M1, U1 = earlier
    return M2@M1, M2@U1+U2
 
aggregate = (np.eye(dk), np.zeros((dk, dv)))
for transition in transitions:
    aggregate = compose(transition, aggregate)
np.testing.assert_allclose(aggregate[0]@S0+aggregate[1], S, atol=1e-12)
left = compose(transitions[2], compose(transitions[1], transitions[0]))
right = compose(compose(transitions[2], transitions[1]), transitions[0])
for x, y in zip(left, right):
    np.testing.assert_allclose(x, y, atol=1e-12)
k, v, a, b = K[0], V[0], A[0], B[0]
wrong = np.diag(a)@(np.eye(dk)-b*np.outer(k, k))@S0+b*np.outer(k, v)
assert np.max(np.abs(wrong-kda_step(S0, k, v, a, b))) > 1e-3
# Analytic derivative of sum(S_next^2) with respect to every old-state entry.
M, U = transitions[0]
expected = 2*M.T@(M@S0+U)
fd = np.zeros_like(S0)
for idx in np.ndindex(S0.shape):
    plus, minus = S0.copy(), S0.copy()
    plus[idx] += 1e-6
    minus[idx] -= 1e-6
    fd[idx] = (np.sum(kda_step(plus,k,v,a,b)**2)-
               np.sum(kda_step(minus,k,v,a,b)**2))/(2e-6)
np.testing.assert_allclose(fd, expected, rtol=1e-6, atol=1e-8)
 
def softmax(z):
    ex = np.exp(z-np.max(z, axis=-1, keepdims=True))
    return ex/ex.sum(axis=-1, keepdims=True)
 
def rms(x, eps=1e-8):
    return x/np.sqrt(np.mean(x*x, axis=-1, keepdims=True)+eps)
 
sources = np.array([[1., 0.], [0., 2.]])
w = np.array([1., 0.])
weights = softmax(rms(sources)@w)
out = weights@sources
np.testing.assert_allclose(out, [0.80442968, 0.39114064], atol=1e-7)
assert not np.allclose(weights@rms(sources), out)
assert not np.allclose(softmax(sources@w)@sources, out)
# A fixed layer pseudo-query need not yield the same weights at another position.
assert not np.allclose(softmax(rms(sources[::-1])@w), weights)
 
# Small full-width shared path plus selected latent experts.
x = rng.normal(size=4)
Wd, Wu = rng.normal(size=(2, 4)), rng.normal(size=(4, 2))
expert_matrices = rng.normal(size=(4, 2, 2))
shared_matrices = rng.normal(size=(2, 4, 4))
raw_scores = np.array([.2, .8, .4, .6])
bias = np.array([.0, -.3, .4, .0])
selected = np.argsort(-(raw_scores+bias), kind='stable')[:2]
mixture = raw_scores[selected]/raw_scores[selected].sum()
z = Wd@x
u = sum(p*(expert_matrices[j]@z) for p,j in zip(mixture,selected))
y = sum(E@x for E in shared_matrices)+Wu@rms(u)
assert y.shape == (4,) and np.isfinite(y).all()
np.testing.assert_array_equal(selected,
    np.argsort(-(raw_scores+bias+10), kind='stable')[:2])
assert not np.allclose(mixture, softmax((raw_scores+bias)[selected]))
 
def situ(g, up):
    sigmoid = np.exp(-np.logaddexp(0., -g))
    return 4*np.tanh(g/4)*sigmoid*25*np.tanh(up/25)
values = situ(np.linspace(-1000,1000,1001), np.linspace(1000,-1000,1001))
assert np.max(np.abs(values)) <= 100
assert abs(situ(.001,.002)-(.001/(1+np.exp(-.001)))*.002) < 1e-12
# The post-projection bound is not100: a large matrix can amplify bounded values.
assert 3*abs(situ(1000.,1000.)) > 100
 
def next_bias(scores, old_bias, k):
    m, n = scores.shape
    if not 0 < k < n or (m*k) % n:
        raise ValueError('fixture requires integer target load and k<n')
    cutoff = np.sort(scores+old_bias, axis=1)[:, -(k+1)]
    margins = scores-cutoff[:, None]
    q = m*k//n
    # (q+1)th largest, strict >, under the no-ties derivation.
    threshold = np.sort(margins, axis=0)[-(q+1)]
    new = -threshold
    return new-new.mean(), margins, threshold
 
scores = rng.uniform(size=(8,4))
nb, margins, threshold = next_bias(scores, np.zeros(4), 1)
fixed_counts = (margins > threshold).sum(axis=0)
assert (fixed_counts <= 2).all()
assert (fixed_counts < 2).any()  # repeated cutoff margins expose the no-ties assumption
no_ties = np.arange(8.)
assert (no_ties > np.sort(no_ties)[-3]).sum() == 2
# Counterexample: identical future tokens choose the same expert under fixed bias.
future = np.tile([.9,.8,.2,.1], (8,1))
counts = np.bincount(np.argmax(future+nb, axis=1), minlength=4)
assert counts.max() == 8 and not np.array_equal(counts,[2,2,2,2])
 
# Exact polar factors illustrate per-head geometry, not Newton-Schulz Muon training.
def polar(matrix):
    u, _, vt = np.linalg.svd(matrix, full_matrices=False)
    return u@vt
momentum = rng.normal(size=(4,3))
whole = polar(momentum)
per_head = np.concatenate([polar(momentum[:2]), polar(momentum[2:])])
assert not np.allclose(whole, per_head)
for head in [per_head[:2], per_head[2:]]:
    np.testing.assert_allclose(head@head.T, np.eye(2), atol=1e-12)
 
# Teacher log-ratios and speculative acceptance overlap are different quantities.
p, q = np.array([.6,.3,.1]), np.array([.2,.3,.5])
reward = np.clip(np.log(p/q), -.5, .5)
assert reward[0] == .5 and reward[2] == -.5
acceptance = np.minimum(p,q).sum()
np.testing.assert_allclose(acceptance, 1-.5*np.abs(p-q).sum())
np.testing.assert_allclose(np.sum(q*np.minimum(1.,p/q)), acceptance)
assert np.isclose(-np.log(acceptance), -np.log(.6))
residual = np.maximum(p-q,0)/(1-acceptance)
np.testing.assert_allclose(np.minimum(p,q)+(1-acceptance)*residual, p)
assert not np.allclose(q, p)  # accepting every draft token would change the target
 
def replay(start, lo, hi):
    state = start.copy()
    for t in range(lo, hi):
        state = kda_step(state, K[t], V[t], A[t], B[t])
    return state
prefix = replay(S0,0,3)
cache = {'boundary':3, 'state':prefix.copy(), 'tokens':V[:3].copy()}
snapshot = cache['state'].copy()
# Three speculative tokens proposed, only two accepted.
all_drafts = replay(cache['state'],3,6)
accepted = replay(cache['state'],3,5)
np.testing.assert_allclose(accepted, replay(S0,0,5))
assert not np.allclose(all_drafts,accepted)
np.testing.assert_array_equal(cache['state'], snapshot)
token_cache = np.concatenate([cache['tokens'],V[3:5]])
assert len(token_cache) == 5
assert cache['boundary'] == len(cache['tokens'])
print('KDA scalar/scan/gradient, AttnRes, MoE, SiTU, quantile, polar, OPD, acceptance and replay pass')
print('KDA old-state finite-difference max error:', np.max(np.abs(fd-expected)))

Worked post-training distinction. If teacher probability is.6 and student probability.2, the unclipped log-ratio is log3≈1.099; with bound.5 the token reward is.5. A detached reward does not backpropagate through the teacher or through its own student-probability construction. That is not the same as minimizing an ordinary differentiable KL expression. A complete RL implementation also needs the policy objective, sampling probabilities, masks, reward normalization and stale-policy treatment; this worksheet reproduces none of Kimi's training outcomes.

Acceptance derivation. A token drawn from draft q is accepted with min(1,p/q), so expected acceptance is sum min(p,q)=1−TV(p,q). On rejection, sampling from normalized positive residual (p−q)+ restores target p. The code proves this identity on three categories and detects the incorrect always-accept rule. It does not measure speedup, prescribe a fixed draft length or exercise a production EAGLE kernel.

Local-reference extension. To study an actual implementation later, pin a FlashKDA revision, build in a compatible environment you are authorized to use, start with a tiny random tensor, and compare sequence outputs and gradients to the scalar recurrence with stated dtype/tolerance. Then test chunk boundaries, cached continuation, rejected-prefix replay and long-tile numerical range. Record accelerator/kernel/compiler versions and warmup/timing conventions before a throughput comparison. The examples here do not build FlashKDA, install its dependencies, download weights, run GPU kernels or benchmark a model.

Pinned evidence and execution boundary

The architecture and equations above are checked against the official report at commit 3cb39dfd32e51c3328e2e4b4af21341247d06c43, especially sections 2.1–2.4, and the matching model summary. These are publisher-reported facts, not independent validation of the weights or benchmark outcomes. The report describes 104B activated parameters, 93 layers, 69 KDA plus 24 MLA, 896 routed experts with 16 selected, and two shared experts. The final extra MLA layer explains why 93 is not divisible by four.

No weights were downloaded. No Kimi inference, multimodal model execution, training, agent benchmark, GPU kernel, distributed system or deployment was run. Exact small NumPy tests establish only their stated algebraic properties. Case studies remain research-audit prompts rather than verified achievements.

Appendix: how to read the figures and source formatting

Use the diagrams to trace tensor shapes and state transitions, then connect each arrow to the defined equation symbols and worked examples. The glossary and prerequisite reading route can help when a mechanism is unfamiliar. Diagram colours and component styles are presentation choices, not architectural properties. If you adapt the article’s MDX source, represent literal quotes and braces according to their MDX/JSX context and check the resulting headings and code blocks; backslash-escaped JSX attributes do not have a universal silent-disappearance rule.