Kimi K3 — Open Frontier Intelligence, Explained From Scratch

Sources (please open these too):

• Moonshot AI blog: kimi.com/blog/kimi-k3
• Technical report (PDF): github.com/MoonshotAI/Kimi-K3
• Model weights: huggingface.co/moonshotai/Kimi-K3
• Related open-source projects referenced: FlashKDA, MiniTriton, nano-KPU, AgentENV
• All figures/images referenced in this post live in the original blog page — open the Moonshot blog in a second tab to see them next to each explanation.

One-line summary. Kimi K3 is a 2.8-trillion-parameter Mixture-of-Experts language model with native vision and a 1-million-token context window, released open-weights by Moonshot AI. It is the first open model in the "3-trillion-parameter class," and its main technical contributions are a new attention mechanism (Kimi Delta Attention), a new way of routing information across layers (Attention Residuals), and a stabilized extreme-sparsity Mixture-of-Experts (Stable LatentMoE) that together give it ~2.5× the scaling efficiency of Kimi K2.

That sentence is dense. The rest of this post unpacks every term in it — starting from "what is attention" — so that by the end you can read the actual tech report and understand what each equation is doing and why.

---

0. How to read this post

This post is written as a layered explainer. Each big idea is introduced three times:

1. Story mode — plain English, no math, why we care.
2. Diagram mode — a picture with labeled boxes.
3. Technical mode — the actual equation or algorithm, with every symbol defined.

If a section gets hairy, you can safely skip the "Technical mode" boxes and still follow the story. If you only care about the math, jump to the technical boxes.

L3

Story mode

one-paragraph plain English

L2

Diagram mode

boxes and arrows

L1

Technical mode

equations with every symbol defined

core

---

1. The one-page TL;DR

L9

2.8T total parameters, 104B activated per token

Mixture-of-Experts sparsity ≈ 27×

L8

1,000,000-token context window

long-horizon coding + agentic work

L7

Native vision (MoonViT-V2 encoder)

images + video in the same model

L6

Kimi Delta Attention + Gated MLA (hybrid)

linear-time long-range mixing + occasional full attention

L5

Attention Residuals (AttnRes)

every layer can read every earlier block

L4

Stable LatentMoE: 16 of 896 routed experts

SiTU-GLU + RMSNorm + Quantile Balancing

core

L3

MXFP4 weights / MXFP8 activations from SFT onward

quantization-aware training for cheap serving

L2

RL across coding, agent, knowledge domains

unified white-box RL environment + on-policy distillation

L1

~2.5× scaling efficiency vs. Kimi K2

more intelligence per unit of compute

The reported headline results (from Figure 1 of the tech report): Kimi K3 sits behind Claude Fable 5 and GPT-5.6 Sol on most benchmarks, but is ahead of every other open and proprietary model tested — including GLM-5.2, GPT-5.5, and Claude Opus 4.8. On some benchmarks (BrowseComp, AutomationBench, ProgramBench, SWE-Marathon), it is the top-scoring model.

Now let’s actually understand why those numbers came out that way.

---

2. Background — what is a language model even doing?

Skip this section if you already know the basics of transformers.

2.1 Language modeling in one sentence

A language model is a function that reads a sequence of tokens (roughly, "word pieces") and predicts the probability distribution of the next token. Everything else — chatting, coding, "reasoning" — is downstream of this single objective.

📥

Tokens in

"The cat sat on the"

Embeddings

vectors of numbers

N transformer blocks

attention + FFN, repeated

Logits

score per vocab token

📤

Next token

"mat"

1. 1
   Tokens in

   "The cat sat on the"
2. 2
   Embeddings

   vectors of numbers
3. 3
   N transformer blocks

   attention + FFN, repeated
4. 4
   Logits

   score per vocab token
5. 5
   Next token

   "mat"

Everything hard about scaling a language model is (a) how the "N transformer blocks" mix information along the sequence, and (b) how much useful compute you can pack into each block.

2.2 The three axes of scale

The Kimi team frames K3 as scaling three complementary dimensions:

L3

Sequence length

how far back the model can look — K3 pushes to 1M tokens via KDA + Gated MLA

L2

Depth

how many layers of processing — AttnRes lets every layer read every earlier block

L1

Width

how many parallel specialists — Stable LatentMoE grows the expert pool to 896

core

Keep this triangle in mind. Every architectural choice below is aimed at one of these three axes.

2.3 Standard attention, in one paragraph

Given a sequence of token vectors, self-attention lets each token pull information from every other token by (i) projecting each vector into three roles — Query (Q), Key (K), Value (V), (ii) computing a similarity score between each Query and every Key, (iii) softmax-normalizing those scores, and (iv) using them as weights to average the Values.

$$\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right) V$$

The problem: computing $QK^\top$ for a sequence of length $L$ costs $\mathcal{O}(L^2)$ time and memory. At $L = 1{,}000{,}000$ this is a trillion pairwise comparisons per layer per head. You cannot brute-force your way to million-token context. This is why Kimi K3 needs a smarter attention.

---

3. Architecture overview

Kimi K3's block is unusual. Each block contains three KDA layers followed by one Gated MLA layer, and each attention layer is paired with a Stable LatentMoE feed-forward network. Above that, Attention Residuals rewire how blocks talk to each other, and at the bottom MoonViT-V2 pipes vision into the same embedding space.

Read that diagram carefully — it is the whole model in one picture. The next four sections zoom into each colored region.

---

4. Kimi Delta Attention (KDA) — linear-time attention with a "forget gate"

Story mode. Standard attention is $\mathcal{O}(L^2)$. Linear attention is a family of tricks that reduces this to $\mathcal{O}(L)$ by rewriting attention as a recurrence: instead of comparing every token to every other token, you maintain a compact "state matrix" that summarizes everything you have seen so far, and you update it one token at a time. KDA is Moonshot's tuned version of one of these tricks, called the delta rule, plus a channel-wise forget gate.

If that felt fast, don't worry — we'll unpack every word.

4.1 Why linear attention exists at all

The classical result behind linear attention is that if you replace the $\text{softmax}$ in attention with a kernel $\phi$ that factorizes, you can rewrite

$$o_t = \sum_{i \le t} \phi(q_t)^\top \phi(k_i)\, v_i^\top = \phi(q_t)^\top S_t, \qquad S_t = S_{t-1} + \phi(k_t) v_t^\top.$$

Read this slowly. The right-hand side says: to compute the output at token $t$, you don't need every previous key and value — you only need $S_t$, a $d_k \times d_v$ matrix that has been rolled forward one token at a time. Each new token adds $\phi(k_t)v_t^\top$ to $S_t$. That is a recurrent neural network, not a transformer.

Cost: $\mathcal{O}(L \cdot d_k d_v)$ — linear in sequence length. Memory: constant per layer.

Problem with vanilla linear attention: because $S_t$ just accumulates forever, the model has no way to "forget" old information. Long sequences drown in stale context.

4.2 The delta rule — writing over yourself instead of adding

The delta rule (Schlag et al., 2021) fixes this by making the recurrence look like an error-correcting write:

$$S_t = \left(I - \beta_t k_t k_t^\top\right) S_{t-1} + \beta_t k_t v_t^\top.$$

The scalar $\beta_t \in (0, 1)$ is a per-token "write strength." Read this as: "erase whatever the current key would have retrieved from the old state, then write in the new value." It behaves much more like a mutable dictionary and less like an infinite accumulator.

4.3 KDA's contribution — channel-wise forgetting

KDA (Kimi Team, Kimi Linear, 2025) adds one more piece: a channel-wise decay vector $\alpha_t \in (0, 1)^{d_k}$. Each channel (each row of the state) is multiplied by its own retention factor before the delta rule fires. This lets the model forget some dimensions of the state faster than others.

$$\boxed{\; S_t \;=\; \left(I - \beta_t k_t k_t^\top\right)\,\text{Diag}(\alpha_t)\, S_{t-1} \;+\; \beta_t k_t v_t^\top, \quad \tilde o_t = S_t^\top q_t. \;}$$

The Kimi K3 report calls this Equation (1). Every symbol:

• $x_t \in \mathbb{R}^d$ — the hidden state of token $t$.
• $q_t, k_t \in \mathbb{R}^{d_k}$ — query and key, both derived from $x_t$.
• $v_t \in \mathbb{R}^{d_v}$ — value, also from $x_t$.
• $S_t \in \mathbb{R}^{d_k \times d_v}$ — the recurrent state (small, fixed-size).
• $\alpha_t$ — per-channel retention (learned, per-token, per-head).
• $\beta_t$ — per-token write strength.
• $I - \beta_t k_t k_t^\top$ — an "erase" projection along the current key.

The projections use ShortConv → Swish → L2Norm, which are just a small 1D convolution, a smooth nonlinearity, and vector-length normalization. These make the queries and keys more stable numerically.

4.4 Why Kimi K3 changed the log-decay parameterization

The tricky part in software is computing $S_t$ in parallel over a chunk of tokens, not one at a time. To do that, you need cumulative products of the decays, and those products can become astronomically small or large — overflowing in fp16/bf16.

Kimi Linear (K2) mapped the raw decay logit $z$ through a negative-softplus: $g = -e^A \, \text{softplus}(z)$. This can go to $-\infty$, so the reciprocal $1/\gamma$ can overflow. Their fix was expensive: split each chunk into 16-token tiles and do the diagonal tiles with explicit position-pair loops that cannot use tensor cores.

Kimi K3 replaces this with a scaled sigmoid that bounds the log-decay from below:

$$g_t^h \;=\; g_{\min}\, \text{Sigmoid}(e^{A_h} z_t^h) \in (g_{\min}, 0)^{d_k}, \qquad g_{\min} = -5.$$

The payoff: the whole KDA kernel now runs as dense matmul on Tensor Cores. That is why K3 gets ~2.5× the effective throughput of K2.

4.5 Chunkwise parallel form (skip if math-averse)

To train efficiently you need to process a chunk of $C$ tokens in parallel. The trick is to compute the intra-chunk contribution as a masked matmul, and the inter-chunk contribution as a state carried between chunks. The tech report’s Equation (4) is:

$$O_{[t]} \;=\; \underbrace{(\Gamma^{1\to C}_{[t]} \odot Q_{[t]})\, S_{[t]}}_{\text{inter-chunk}} \;+\; \underbrace{A_{[t]}\, \widetilde V_{[t]}}_{\text{intra-chunk}}, \quad A_{[t]} = \text{Tril}\!\left[(Q_{[t]}\!\odot\!\Gamma^{1\to C}_{[t]})(K_{[t]}/\Gamma^{1\to C}_{[t]})^\top\right].$$

The Tril mask enforces causality (a token can only attend to itself and earlier tokens). The bounded decay guarantees the $1/\Gamma$ division does not blow up.

4.6 Gated MLA — the "occasional global attention"

KDA is fast but its state is a fixed-size matrix, which limits precision on very hard lookup tasks. So K3 interleaves one Gated MLA layer for every three KDA layers. MLA = Multi-head Latent Attention (from DeepSeek-V2), which is standard $\mathcal{O}(L^2)$ attention but with a compressed key/value cache. "Gated" adds a learned gate so the attention output can be attenuated when it isn't useful.

The intuition: KDA is a fast summarizer; Gated MLA is a precise retriever. Three summarizers per retriever is a compute-quality sweet spot Moonshot found empirically.

---

5. Attention Residuals (AttnRes) — every layer talks to every earlier block

Story mode. In a normal transformer, information flows straight up the stack via residual connections: layer $\ell$'s input is layer $\ell{-}1$'s output. That means information from layer $3$ has to be re-encoded through layers $4, 5, \ldots$ if a later layer wants to use it. Attention Residuals let a later layer directly attend to the outputs of any earlier block — like a shortcut network across depth.

5.1 Standard residuals vs. attention residuals

5.2 The AttnRes equations

Each layer stores a learned pseudo-query $w$. That query attends over the block outputs $[b_0, b_1, \ldots, b_{n-1}]$ (where $b_0$ is the embedding), producing attention weights $\alpha$. The residual added to the current layer's output is a weighted mix of those past block outputs.

$$\alpha \;=\; \text{softmax}(w^\top B),\qquad \text{residual} \;=\; B\, \alpha, \qquad B = [b_0, b_1, \ldots, b_{n-1}].$$

Crucially:

• Attention weights $\alpha$ are shared per position and there is one $w$ per layer, so this adds negligible parameters.
• Attention is over blocks (chunks of layers), not individual layers, keeping memory at $\mathcal{O}(Nd)$ where $N \approx 8$ is the number of blocks.
• Empirically $N \approx 8$ recovers most of the benefit; K3 uses 8 blocks of 12 layers each ≈ 96 layers total.

Why it matters. Deep networks tend to "forget" early features because gradient signals attenuate. AttnRes gives each layer explicit read access to earlier abstractions — a bit like a residual highway plus selective routing. In the tech report, ablations show AttnRes contributes a meaningful chunk of the 2.5× scaling-efficiency win.

---

6. Stable LatentMoE — 16 experts out of 896

Story mode. A "Mixture of Experts" replaces a single big feed-forward network in each block with many small ones ("experts"), plus a router that picks which few to run for each token. The result: total parameters go up (more knowledge storage), but compute per token stays flat (only a handful of experts activate). Kimi K3 pushes this to an extreme — 896 routed experts, 16 activated per token, sparsity ratio 56. That is roughly 3–4× more expert diversity than any published open model.

Extreme sparsity is hard for two reasons:

1. Activation explosion. Chained matmuls in the routed branch amplify outliers.
2. Load balancing. With 896 experts, the router will inevitably neglect some and overload others.

Stable LatentMoE fixes both. Let's take them one at a time.

6.1 LatentMoE — cheap experts via a shared bottleneck

Vanilla MoE hands each expert the full $d$-dimensional hidden state, and each expert has its own $d \to 4d \to d$ MLP. If you activate 16 experts, you pay 16× the communication and expert weight traffic per token.

LatentMoE (Elango et al., 2026) splits the model width into two pieces:

• A shared full-width path (2 shared experts in K3) that always runs.
• A compact latent path at width $\ell \ll d$: a shared projection $W^\downarrow$ takes the token down to $\ell$; each routed expert operates in $\mathbb{R}^\ell$; a single $W^\uparrow$ maps the aggregated result back to $\mathbb{R}^d$.

$$u \;=\; \sum_{i \in \mathcal T_k(x)} p_i\, E_i^{\text{routed}}(W^\downarrow x), \qquad y \;=\; \sum_{j=1}^{N_s} E_j^{\text{shared}}(x) \;+\; W^\uparrow\, \text{RMSNorm}(u).$$

This is Equation (11) in the tech report. It is the payoff of the whole design: you can now afford 896 routed experts because they operate in a narrow latent space.

6.2 Normalized LatentMoE — one extra RMSNorm

The routed aggregate $u$ can have wildly varying scale depending on which experts were picked and their weights. K3 inserts an RMSNorm before the up-projection. RMSNorm (Zhang & Sennrich, 2019) is just: divide the vector by the root-mean-square of its entries, then scale by a learned gain. This one line materially improves validation loss.

6.3 SiTU-GLU — a bounded activation

The classic transformer FFN uses SwiGLU: $\text{SwiGLU}(x) = (x \cdot \sigma(W_g x)) \odot (W_u x)$. Both factors are unbounded, so if a "gate" and "up" projection happen to spike together on one channel, you get huge activations — which overflow in low-precision arithmetic (a real problem for MXFP4/FP8 training).

K3 introduces SiTU-GLU (Sigmoid Tanh Unit GLU), which "smoothly caps" both branches with a scaled tanh:

$$\text{SiTU-GLU}(x) \;=\; \beta_1 \tanh\!\left(\frac{W_g x}{\beta_1}\right) \odot \sigma(W_g x) \odot \beta_2 \tanh\!\left(\frac{W_u x}{\beta_2}\right)$$

with $\beta_1 = 4$ and $\beta_2 = 25$. Two nice properties:

• Near the origin, $\beta \tanh(z/\beta) \approx z$ — so SiTU behaves like SwiGLU on typical inputs.
• Far from the origin, both branches saturate. The output is bounded: $\|\text{SiTU-GLU}(x)\|_\infty \le \beta_1 \beta_2 = 100$.

6.4 Quantile Balancing — solving load balancing without an auxiliary loss

The problem. In MoE, some experts get overwhelmed with tokens ("hot"), others starve ("dying"). With 896 experts, this is a serious production risk: hot experts slow down training, dying experts waste parameters.

Prior state of the art (DeepSeek-V3) added a per-expert bias $b_j$ to the router score, and nudged $b_j$ by $\pm \gamma$ at every step depending on whether expert $j$ was over- or under-utilized. Simple. But at 896 experts the fixed step size $\gamma$ becomes very hard to tune: too small → slow to adapt, too large → oscillates.

K3's idea. Solve for the bias exactly. Every step, compute the histogram of router scores per expert, and set the bias so that exactly the target number of tokens will route to each expert.

Concretely (Equation 14):

$$\hat b_j^{(t+1)} \;\leftarrow\; -\,\text{quantile}_{\,1 - k/n}\!\left(s_{:,j}^{(t)} - \alpha^{(t)}\right), \qquad b^{(t+1)} \;\leftarrow\; \hat b^{(t+1)} - \text{mean}(\hat b^{(t+1)})\, \mathbf 1,$$

where the "margin" $s_{i,j} - \alpha_i^{(t)}$ is the biased router score minus the "cutoff" the token needs to enter its top-$k$. The bias for expert $j$ is set to whatever value would put exactly $mk/n$ tokens above the cutoff — this is a quantile of the margin distribution.

Why it is practical. At scale, "computing the exact quantile" over billions of margins is impossible. K3 uses a histogram approximation: each rank bins its local margins into $\sim 8000$ buckets per expert, a single all-reduce sums the bin counts, and the quantile is read off the pooled histogram. Total extra cost: one small collective per step. No auxiliary loss, no tunable step size, and the bias is frozen at inference.

6.5 Per-Head Muon — one optimizer detail worth mentioning

Muon is a recent optimizer for hidden-layer weights (Jordan et al., 2024, scaled by Moonshot in 2025) that updates whole weight matrices with a specially-normalized step. K3 extends this to treat each attention head as its own optimization group, which stabilizes training when head widths differ or when some heads specialize faster than others.

---

7. Vision — MoonViT-V2

Kimi K3 is natively multimodal — vision is not bolted on after training. Images and video frames go through MoonViT-V2, a vision transformer trained jointly with the language model. A small MLP projector maps its visual features into the same embedding space the text tokens live in.

🖼️

Pixels

image or video frame

MoonViT-V2

patch → tokens

MLP proj

align to text space

Embedding

unified sequence

K3 backbone

1. 1
   Pixels

   image or video frame
2. 2
   MoonViT-V2

   patch → tokens
3. 3
   MLP proj

   align to text space
4. 4
   Embedding

   unified sequence
5. 5
   K3 backbone

Practical consequence: the model can, mid-reasoning, execute code to zoom or crop an image, feed the result back as a new visual observation, and continue — the "vision-in-the-loop" pattern the blog highlights for game-dev and CAD tasks.

---

8. Training — what actually happened

8.1 Pre-training

• Scale. 2.8T parameters, 104B active, MoE sparsity ≈ 27.
• Data. Refined recipes over Kimi K2; the report emphasizes better data curation contributed to the 2.5× win alongside architecture.
• Efficiency. Kernel-level co-design: fused kernels for KDA, sequence parallelism for AttnRes, custom kernels for the latent MoE.

8.2 Post-training pipeline

📚

SFT

supervised fine-tune on curated demos

Domain RL

coding / agent / knowledge / vision, each at multiple effort levels

Multi-teacher distillation

consolidate specialists into one unified model

1. 1
   SFT

   supervised fine-tune on curated demos
2. 2
   Domain RL

   coding / agent / knowledge / vision, each at multiple effort levels
3. 3
   Multi-teacher distillation

   consolidate specialists into one unified model

8.3 Reinforcement Learning across "effort levels"

The team trains the model at multiple reasoning-effort levels ("low", "high", "max") within each domain, then merges them. This is what lets the same K3 model spend a few hundred tokens on an easy question or millions on a research report — without regressing on either.

8.4 On-policy distillation with dense rewards

Instead of the usual "sparse verifier" reward, K3 uses a dense token-level KL-style reward comparing the student to a domain-specialist teacher:

$$r_{\text{opd}}(y_t \mid e, x, y_{<t}) \;=\; \text{clip}\!\left(\text{sg}\log\frac{\pi_{\text{teacher}}^{(d,e)}(y_t)}{\pi_\theta(y_t)},\ -R_{\max},\ R_{\max}\right).$$

Every symbol: $e$ = effort level, $d$ = domain, $\text{sg}$ = stop gradient, $R_{\max}$ = clipping threshold to prevent extreme advantages. Every generated token gets a reward — much denser signal than "success/failure at the end of a trajectory," which is why long-horizon tasks (thousands of tool calls) actually train.

8.5 Unified white-box RL environment

Every "agent harness" (Kimi Code, Claude Code, Codex, OpenClaw, Hermes) is decomposed into interchangeable modules: tools, prompts, context management, memory, subagents. During RL, tasks are matched with randomized harness configurations. This prevents overfitting to any single harness's conventions — the resulting policy generalizes to your harness.

8.6 Knowledge-graph-guided task synthesis

Post-training tasks are only as good as their source material. Moonshot builds a self-evolving, hierarchical knowledge graph (agents recursively explore and add nodes), then samples related nodes as keyword sets to guide web retrieval and task synthesis.

---

9. Infrastructure — how they actually ran this

The tech report spends the most pages here, because at 2.8T parameters, infrastructure is the model.

9.1 MoonEP — perfectly balanced expert parallelism

Expert parallelism means: different GPUs hold different experts. If token routing is imbalanced, some GPUs sit idle. MoonEP enforces static computation shapes (each GPU processes the same fixed number of tokens per step) with zero-copy communication. Combined with Quantile Balancing, this eliminates the biggest source of MoE training stalls.

9.2 KDA Context Parallelism

For 1M-token sequences, no single GPU can hold the whole context. Context parallelism shards the sequence across GPUs and communicates the KDA recurrent state at chunk boundaries. K3 pairs this with state-aware prefix caching so that if two requests share a long prefix (very common in coding sessions), the KDA state at the shared boundary can be restored from cache instead of recomputed.

9.3 KDA-aware prefix cache — the deployment story

Normal LLM serving caches the "KV cache" of long prefixes so a second request with the same prefix skips prefill. For a 1M-token coding session with a 400K-token prefix, this is life-or-death for latency.

K3 has two caches per block: the MLA KV cache (grows per token) and the KDA recurrent state (fixed size, one per request). A cached prefix is only reusable if both are restorable at the same boundary. K3’s design:

Consequence: the announced API pricing — $0.30/MTok cache-hit input** vs. **$3.00/MTok cache-miss input — is a direct consequence of this design. Coding workloads hit the cache >90% of the time.

9.4 AgentENV — resumable microVM sandboxes for agentic RL

RL for a coding agent needs an environment where the agent can safely run arbitrary code. Containers are cheap but leak; VMs are safe but slow. AgentENV uses Firecracker microVMs with incremental checkpointing:

• Checkpoint latency: 133 ms
• Resume latency: 49 ms
• Pause/Resume: paused sandboxes cost 0 CPU/memory
• Fork: create a side-effect-free copy for reward judging
• Snapshot: for recovery

During K3 training and evaluation, 51.2 million sandboxes were created across 1.5 million distinct images. AgentENV is open-sourced here.

9.5 Fleet-level scheduling

Two policies keep production serving predictable:

• Cache-aware affinity scheduling. Each session pinned to the cluster holding its cache (via consistent hashing), with a hot-standby cluster for failover.
• Budget-based admission control. Each request class (short vs. million-token) gets its own resource budget so bursty long-context traffic can't starve short requests.

---

10. Deployment — how it stays cheap

10.1 MXFP4 quantization-aware training

MXFP4 is a 4-bit floating point format (Rouhani et al., 2023). K3 quantizes MoE expert weights (which dominate memory) to MXFP4, with activations at MXFP8, keeping attention and shared experts in higher precision. Quantization-Aware Training (QAT) is used throughout SFT and RL — so the model is trained with the exact quantization scheme it will be served under. No train/serve mismatch.

L3

Attention projections, MLA, router, shared experts

BF16 / FP16

L2

MoE expert activations

MXFP8

L1

MoE expert weights

MXFP4 (4-bit)

core

10.2 EAGLE-3 speculative decoding

K3's multi-token-prediction (MTP) head is fine-tuned into an EAGLE-3 draft model: a small network that predicts the next 7 tokens speculatively, which the full model then verifies in parallel. Instead of the standard KL loss, the draft is trained directly on the LK loss:

$$\mathcal L_{LK} \;=\; -\log \sum_{x \in V} \min\!\bigl(p(x),\, q(x)\bigr)$$

which is the negative log of the acceptance rate itself. This gives a measurable throughput win at inference.

10.3 KDA state replay for speculative decoding

A subtle problem: if the draft's 7 tokens include some that are rejected, the KDA state has already advanced. Rolling back a large state per step is expensive. K3 caches only the projected inputs of drafted tokens (much smaller than the state), then replays the accepted prefix on-chip in a single fused kernel. This is the concurrent-work "ReplaySSM" idea, integrated cleanly into the serving stack.

---

11. Benchmarks — what they mean

Take these with the usual grain of salt (max reasoning effort, harness-dependent), but Moonshot at least documented the harness they used per benchmark in the blog’s footnotes.

How to read each benchmark:

• DeepSWE — software-engineering issues from real GitHub repos; the model is scored on whether its patch passes tests.
• Terminal-Bench 2.1 — realistic command-line tasks; the harness runs commands and grades outcomes.
• FrontierSWE / SWE-Marathon / KCB 2.0 / ProgramBench — variations on "solve real bugs / write real code."
• BrowseComp — an agent has to browse the web and produce a verifiable answer. K3 is #1 here (with context compaction at 300K).
• AutomationBench / JobBench / AA-Briefcase — human-graded agentic knowledge-work benchmarks (finance, legal, ops).
• CharXiv / ZeroBench / MMMU-Pro / PerceptionBench — vision-heavy reasoning.

The pattern to notice: K3 wins on long-horizon agentic tasks (BrowseComp, AutomationBench, SWE-Marathon) — exactly where its 1M context, KDA cache-hit economics, and RL environments are designed to help.

---

12. Case studies (from the blog)

12.1 MiniTriton — a GPU compiler K3 wrote

Given a spec, K3 built MiniTriton: a Triton-like DSL frontend, MLIR-based IR passes, and a PTX code-generation pipeline, plus a PyTorch-like tensor library with autograd and NCCL primitives. Result: on an NVIDIA L20, MiniTriton beats torch.compile and Triton on geometric mean over its benchmark suite, approaches cuBLAS at large matmul sizes, and trains a GPT model end-to-end with a loss curve nearly identical to PyTorch reference (fp32 rounding-level agreement). Repo: github.com/MoonshotAI/minitriton.

12.2 nano-KPU — a chip K3 designed in 48 hours

K3 designed an inference-chip prototype for a "nano" model built on the same architecture (KDA + NoPE-MLA + Block AttnRes + INT4 MoE). Using open-source EDA tools on the Nangate45 standard-cell library, it produced a 4 mm² design that closes timing at 100 MHz and hits >8,700 tokens/sec decode throughput in RTL simulation. Repo: github.com/MoonshotAI/nano-kpu.

12.3 I–Love–Q astrophysics reproduction

To reproduce a classical result in computational astrophysics (universal relations between neutron-star moment of inertia, tidal Love number, and quadrupole moment), K3 read 20+ papers, implemented the full numerical pipeline, evaluated 300+ equations of state, spotted inconsistencies in published formulas, and produced an interactive HTML dashboard — in ~2 hours vs. weeks for a human researcher.

12.4 GWTC-5 gravitational-wave analysis

391 events analyzed in parallel using 20+ concurrent subagents, producing 7 figures, 2 tables, and a synthesis over 10+ papers.

12.5 Video editing from 56 clips

K3 cut its own teaser video: clip selection, motion-matched cuts, beat-synced timing, audio processing, multiple revisions. Because the model is natively multimodal, "understand the clip" and "write the edit decision list" happen in the same forward pass.

---

13. Limitations (from the blog)

L4

Sensitive to thinking history

if the harness drops <think> traces or you swap models mid-session, quality degrades sharply

L3

Excessive proactiveness

on ambiguous requests it may 'just decide' — needs explicit boundaries in AGENTS.md

L2

UX gap vs Fable 5 / GPT-5.6 Sol

raw capability is competitive; polish and refusal behavior lag

L1

Still trails top proprietary models overall

on the hardest coding + reasoning benchmarks

core

The first two are worth reading twice. K3 was trained in preserved-thinking-history mode, which means the RL policy assumes its own reasoning traces come back in the next turn. Standard OpenAI-style APIs that drop <think> blocks between turns can break it. Moonshot recommends the Kimi Code harness for that reason.

---

14. What this actually means for open source

Three things are new about K3:

1. A 3T-class model exists in the open. No open lab had crossed 2T before. K3 pushes the ceiling by ~40%.
2. The scaling advances are architectural, not just "more data / more compute." KDA + AttnRes + Stable LatentMoE are the design of the model, not tricks bolted on.
3. Infrastructure was co-designed and mostly open-sourced. FlashKDA, MiniTriton, nano-KPU, and AgentENV all have public repos. Anyone rebuilding a frontier model can start from these instead of reinventing them.

The gap to Fable 5 / GPT-5.6 Sol is real, but it is a capability gap, not an architectural gap. That is a very different position for open source than "we're one training run behind" — because now the training tools themselves are open.

---

15. Glossary (skim this after)

L16

MoE (Mixture of Experts)

many small FFNs; a router picks which few to run per token

L15

Sparsity ratio

total experts / active experts (K3 = 896/16 ≈ 56)

L14

Linear attention

O(L) attention via a recurrent state; loses precise recall

L13

Delta rule

error-correcting recurrent update; erase-then-write

L12

KDA

delta rule + channel-wise forget gate + bounded log-decay

L11

MLA

Multi-head Latent Attention — full attention with compressed KV cache

L10

AttnRes

learned attention across block outputs — non-sequential residuals

L9

LatentMoE

shared W↓ / W↑ let routed experts run in a compact latent space

L8

SiTU-GLU

smoothly-capped SwiGLU; bounded output for low-precision safety

L7

Quantile Balancing

exact per-expert bias from margin quantiles

L6

Per-Head Muon

Muon optimizer applied independently per attention head

L5

MXFP4 / MXFP8

block-scaled 4- and 8-bit floating point formats

L4

QAT

quantization-aware training — train under serving precision

L3

EAGLE-3

draft-and-verify speculative decoding, trained with LK loss

L2

MoonEP

expert-parallel training with static shapes, zero-copy

L1

AgentENV

resumable microVM sandbox for agentic RL

core

---

16. Reading path if you want to go deeper

1. The blog post first. kimi.com/blog/kimi-k3 has the images and case-study videos we couldn't inline here.
2. The tech report, Section 2 (architecture) — with this post open in a second tab, every equation should now be readable.
3. Kimi Linear paper (arXiv 2510.26692) — the original KDA introduction.
4. DeepSeek-V3 tech report (arXiv 2412.19437) — the auxiliary-loss-free routing K3 improves on, and the MLA baseline.
5. LatentMoE (arXiv 2601.18089) — the paper Stable LatentMoE builds on.
6. Muon optimizer (Jordan et al., 2024) — for Per-Head Muon.
7. EAGLE-3 (arXiv 2503.01840) — for the draft-model design.

---

Appendix — A note on how this post was formatted (reusable guideline)

For future deep-dive posts, this is the format that worked. Keep it.

L10

Sources block at the very top

primary blog + tech report + repos; readers can verify anything you say

L9

One-page TL;DR + headline chart

so a busy reader gets 80% in 60 seconds

L8

Background section (skippable)

for readers who don't have the prereqs — mark as skippable

L7

One diagram per concept

use ArchitectureDiagram / DataFlow / ConceptLayers / XYPlot — no ad-hoc SVGs

L6

Every equation with every symbol defined

no orphan variables, no `where obvious'

L5

Story mode → Diagram mode → Technical mode

three passes per idea, in that order

core

L4

Track color consistency

one track (violet=LLM here) throughout for visual language

L3

Glossary at the end

single skim reference for the terms introduced

L2

Reading path for deeper study

primary sources ordered by prerequisite

L1

MDX escape hygiene

no `<digit`, no bare `{...}` in prose; JSX attrs use `&apos;` or `&quot;` entities

The ' / " rule is not stylistic — backslash escapes inside JSX attributes silently drop the attribute and desync neighboring diagrams. Every future post should follow this or the diagrams will render blank without a build error.