Dissecting the Kimi K3 Technical Report: Moonshot AI Starts Setting Problems for DeepSeek

On the night of July 24, Jensen Huang posted his first tweet on X. It wasn’t about chips: he shared an open letter co-signed by 25 companies, including NVIDIA, Microsoft, Meta, and IBM, advocating for open AI models and urging Congress not to impose premature restrictions.

The same week, the full weights of Kimi K3 landed on Hugging Face.

Ten days earlier, on July 16, K3 had topped the WebDev Arena leaderboard hours after launch, the first time an open model beat every closed model on a blind coding benchmark. Nathan Lambert wrote a line that got quoted repeatedly afterward: “frontier open-weight models are now real.” Kimi had to pause new subscriptions because of compute shortages.

All of this happened before the weights were even released. How did Kimi pull it off? The 47-page technical report just published answers a lot of that.

Right in the introduction, K3 stakes out a position.

Scaling a large model runs along two axes: pre-deployment compute (bigger models, more data) and post-deployment compute (inference time, reinforcement learning, long-horizon agent execution). Over the past two years the open-source community has moved fast on the second axis and barely on the first; most models are still stuck around the 1T-parameter mark. As post-training methods applied to similarly sized base models keep getting more sophisticated, open-source progress will converge with itself, and the gap to the strongest closed models will keep widening.

Then the report cites DeepSeek V4 directly.

DeepSeek V4’s roadmap, laid out just over a month earlier, was to redesign the attention mechanism and training recipe to cut the cost of million-token context, weighted toward the deployment side. K3’s roadmap argues the pre-training axis has to catch up first, that both axes need pushing to the frontier together. The parameter counts diverge sharply: V4 Pro has 1.6 trillion total parameters with 49 billion activated per token, and keeps pushing activation down further. K3 has 2.78 trillion total parameters with 104.2 billion activated per token, more than double V4 Pro’s activation.

The two technical reports end up talking to each other across the distance.

Where the 2.5x Efficiency Comes From

K3’s data shows an overall scaling-efficiency gain of roughly 2.5x over the previous generation, K2.

Where does that come from?

A large model handles information along three axes: how tokens talk to each other within a layer (sequence), how layers pass information to each other (depth), and how many experts each token gets routed to (width).

Standard attention has every word compare itself against every prior word: double the text, quadruple the compute. That makes million-token context essentially uncomputable. K3 replaces three-quarters of its attention layers with a linear-attention variant called KDA, which maintains a single fixed-size state instead of looking back word by word, so compute grows linearly with text length.

The trade-off is that linear attention is less expressive than standard attention, so K3 keeps one standard attention layer (MLA) after every three KDA layers, a 3:1 hybrid. Routine processing runs through KDA; MLA takes over when global context is needed. Qwen3-Next and MiniMax M1 use similar hybrid setups, so this is becoming a fairly standard engineering choice.

K3 also improves on its own earlier work by adding a lower bound to KDA’s decay rate. Without it, intermediate values could grow unboundedly and force the GPU onto a slower code path that couldn’t use the fastest compute units. With the bound in place, everything runs on the fast path, so one parameter tweak unblocks the whole compute chain.

There’s a companion design choice too: K3 drops positional encoding entirely. The conventional approach assigns each word a “seat number” so the model knows what comes before what. K3 skips the seat numbers and lets KDA’s own decay mechanism keep track of position. The direct payoff is that the model can natively handle a million-token context without extra post-hoc adaptation or interpolation engineering.

AttnRes: Letting Layer 93 Read Layer 1’s Notes Directly

KDA handles communication within a layer. K3 has 93 layers, so the question becomes how layers talk to each other.

Deep learning has passed information forward through residual connections for a decade: each layer only sees what the previous layer hands it, with everything earlier compressed into that single state as it moves forward. That works with few layers, but by layer 93, features computed early on have been diluted almost to nothing.

K3’s fix, AttnRes, gives every layer retrieval parameters so it can pull directly from the outputs of any preceding layer. Layer 90 can read layer 1’s raw output, or grab an intermediate result from layer 40, without waiting for it to filter down layer by layer. KDA handles word-to-word communication within a layer; AttnRes handles layer-to-layer communication.

Full retrieval across all 93x93 layer pairs would be too expensive, so K3 groups the 93 layers into seven 12-layer blocks plus one 9-layer tail block, nine blocks total including the embedding layer, and restricts retrieval to those blocks.

DeepSeek has also been reworking this same decade-old structure this year. The approaches differ, but both land on the same conclusion: standard residual connections become a bottleneck at the hundred-layer scale.

Stable LatentMoE: Stabilizing 896 Experts

Sequence direction is handled by KDA, depth direction by AttnRes. The third axis is width, how many experts each token gets routed to.

MoE (Mixture of Experts) places many expert sub-networks inside the model and calls only a few per token, so total parameter count is huge but compute per token stays controlled. K3 pushes this further than most: 896 experts, only 16 used per token, under 2% utilization.

At that scale, two normally minor issues get serious. First, the long compute chain makes intermediate values prone to blowing up. K3 adds normalization layers at key points and swaps in an activation function with a ceiling (SiTU-GLU): normal response for small values, clipping for large ones.

Second, with 896 experts, some get called constantly while others sit idle. The usual fix is a penalty term to rebalance load, but at this scale that penalty term becomes very hard to tune. K3 uses quantile balancing instead: it reads the statistical distribution of scores each token gives each expert and derives each expert’s ideal routing bias from that distribution directly, in one computation. The report includes a proof that this can theoretically reach perfect balance.

This improves directly on DeepSeek V3’s approach, which adjusts with a fixed step size and lags when load fluctuates. Quantile balancing tracks the distribution itself and reacts faster. K3 took DeepSeek’s direction and executed it more precisely.

Infrastructure: 51.2 Million Sandboxes and Nine RL Teachers

The architecture is only half the story; a good design still has to train. Part of K3’s 2.5x gain comes from the training side.

For the vision encoder, standard practice is to initialize from someone else’s pretrained model. K3 trains it from scratch instead, with its own optimizer and data adjustments.

K3’s MoonEP built a load-balancing scheme for parallel training across 896 experts, with a proof that reserving a small number of redundant experts guarantees balanced allocation under any load, so training doesn’t stall. RL training used sandboxes on lightweight virtual machines: 51.2 million created across training and evaluation, with 133ms to snapshot and 49ms to restore. That scale shows how much K3 has invested in the training line that lets the model act autonomously.

Post-training got similar investment. K3 trained three teacher models at different reasoning intensities across three domains (general, agentic, coding), nine teachers total, all built in-house, then distilled back into a single unified model. It hits 91.2% on BrowseComp at half the cost of GPT-5.6 Sol, and scores 4 points below Claude Fable 5 on Kimi Code Bench at 38% of the cost.

A Turning Point in the Open-vs-Closed-Model Momentum

DeepSeek’s fingerprints run through K3’s report. MLA carries over DeepSeek-V2’s attention-compression mechanism, the auxiliary-loss-free routing strategy comes from V3, and LatentMoE’s shared/routed expert split builds on DeepSeek’s research too.

But K3 has also started setting problems for DeepSeek.

The report includes a GPU-kernel optimization case study where the model optimizes four representative kernels, one of which is DSA (DeepSeek Sparse Attention), DeepSeek’s own core attention operator.

K3 cut DSA’s runtime by 55.1%. Having your own core operator sped up by more than half by someone else makes it hard for DeepSeek’s next release not to respond.

K3 has given a fair amount back to open source too. KDA shipped with the FlashKDA kernel and KDA Context Parallelism, so other teams don’t have to build linear attention from scratch. Quantile balancing improves on DeepSeek-V3’s load-balancing scheme. MoonEP’s balance proof advances the same problem DeepEP has been working on. SiTU-GLU and AttnRes are both reusable by other models. K3’s own MiniTriton, a Triton-like GPU compiler, already matches or beats official Triton on core benchmarks on NVIDIA L20, and it’s open-sourced.

Detailed technical reports let the other side build one step further. DeepSeek-V3 dropped the auxiliary loss for MoE; K3 did it more precisely with the quantile method. K3 worked through linear attention’s engineering problems once; the next team that wants KDA has a lot less ground to cover.

The report leaves open questions for other open models too, not just Kimi: will DeepSeek’s next version adopt quantile balancing and a hybrid linear-attention architecture? As open models keep scaling toward larger parameter counts, what will they build on top of the pretraining frameworks Kimi just open-sourced?

Kimi and DeepSeek may not be the last pair whose reports build on each other this way. It’s momentum the open-source community has been building together over the past several years, and it may be the point where the competitive balance between open and closed models starts to reverse.