Kimi K3: The New Heavyweight Champ of Open Weights
What happened
Moonshot AI released Kimi K3, a scaled-up production version of last year's Kimi Linear, jumping from 48B to a staggering 2.8T parameters — making it the largest open-weight model currently available. Architecture watcher Sebastian Raschka broke down the details: a new LatentMoE component borrowed from Nemotron 3 Ultra, plus efficiency-focused upgrades across attention and residual connections.
Why this matters
K3 ditches RoPE entirely in favor of NoPE (no positional embeddings) everywhere — a first for a frontier-scale model, according to Raschka. Combined with attention residuals that link layers via learned attention weights, it signals where the open-weight arms race is heading: not just bigger, but architecturally leaner per-inference-dollar.
The slightly cynical read
Every few months there's a new "biggest open-weight model," and every few months the actual usable-by-mortals version requires a GPU cluster the size of a small country. Scaling from 48B to 2.8T is impressive on a benchmark chart; less impressive when you're the one paying the inference bill.
What to watch next
Watch whether Kimi K3's efficiency tricks (LatentMoE, attention residuals) get copied into the next wave of releases the way DeepSeek's MoE innovations did — and whether anyone actually benchmarks real-world cost-per-token at this scale.