What It Means to Build the World's Largest OpenAI Model
The gap between what proprietary AI labs
release to paying customers and what the broader world can actually access,
study, and build on has been one of the defining tensions in artificial
intelligence. Closed models from the world's best-funded labs set the frontier.
Open models follow, sometimes years behind. Kimi K3 is Moonshot AI's most
direct answer to that gap. Kimi K3 is a 2.8-trillion-parameter model built on
Kimi Delta Attention and Attention Residuals, with native vision capabilities
and a 1-million-token context window. It is the world's first open
3-trillion-class model, designed for frontier intelligence across long-horizon
coding, knowledge work, and reasoning.
That scale matters not as a vanity metric but
as an engineering statement. Kimi K3 marks the latest step in a sustained push
at the scaling frontier where, for nine of the past twelve months, Kimi models
have set the upper bound of open-model sizes. The model's overall performance
still trails the most powerful proprietary systems, Claude Fable 5 and GPT 5.6
Sol, but it demonstrated frontier-level performance across an extensive
evaluation suite, consistently outperforming all other tested models outside that
top tier.
How the Architecture Behind K3 Converts Scale Into Intelligence
A model with 2.8 trillion parameters is only
impressive if the architecture behind it uses that scale efficiently, and this
is where Kimi K3's engineering choices make the real difference. The model is
built on two core architectural innovations: Kimi Delta Attention and Attention
Residuals. Kimi Delta Attention provides an efficient foundation for scaling
attention across long sequences, while Attention Residuals selectively retrieve
representations across model depth rather than accumulating them uniformly,
improving how information flows through the model on tasks that require
sustained reasoning over very long inputs.
On top of those attention innovations, K3 uses
Stable LatentMoE, a Mixture of Experts framework that effectively activates 16
out of 896 experts per token during inference. That extreme sparsity means the
model deploys a fraction of its total parameters on any given task, keeping
inference costs tractable while retaining the full expressive capacity of the
2.8-trillion-parameter architecture. Additional innovations include Quantile
Balancing for expert allocation, Per-Head Muon for adaptive attention optimization,
and quantization-aware training using MXFP4 weights with MXFP8 activations for
broad hardware compatibility. Together with refined training and data recipes,
these structural choices yield approximately 2.5 times better scaling
efficiency compared to Kimi K2, allowing K3 to convert compute into
intelligence significantly more effectively than its predecessor.
What K3 Actually Does That No Open Model Has Done Before
The most compelling evidence for what K3
changes come from its real-world task demonstrations rather than benchmark
numbers alone. In kernel optimization, K3 worked independently in a controlled
sandbox to profile, rewrite, and benchmark GPU kernels across multiple hardware
targets, performing competitively with the best proprietary frontier models on
that task. More strikingly, in the late stages of K3's own development, an
early version of K3 handled the majority of the team's kernel optimization
work, making it among the first AI models to meaningfully contribute to
engineering the next generation of itself.
For GPU compiler development, K3 built
MiniTriton from scratch: a compact Triton-like compiler with its own tile-level
intermediate representation, optimization passes, and PTX code-generation
pipeline. Across supported benchmarks, MiniTriton delivered performance on par
with or better than Triton and Torch. Compile and sustain full end-to-end
neural network training with stable convergence. For chip design, K3 completed
in a single 48-hour autonomous run what typically takes a team of engineers
weeks: designing a chip to serve a nano model built on its own architecture,
using open-source electronic design automation tools. Within a 4 mm² area, the
chip closes timing at 100MHz and sustains over 8,700 tokens per second in
decode throughput simulation. A chip built by a model, for a model, represents
one of the most striking demonstrations of long-horizon agentic capability any
AI system has produced.
How K3 Handles Knowledge Work and Multimodal Tasks
Beyond code and engineering, K3 advances what
Moonshot AI calls end-to-end knowledge work. In one documented case, K3
completed in approximately two hours what would typically require one to two
weeks of work from an experienced researcher: reproducing the I-Love-Q
universal relations in computational astrophysics, which required reviewing and
cross-validating more than twenty papers, implementing a full numerical
pipeline, evaluating more than 300 equations of state, identifying
inconsistencies in published formulas, generating more than 3,000 lines of
Python code, and producing an interactive HTML dashboard for exploring the
results.
The model's native multimodal architecture
processes text, images, and video within the same model rather than through
separate pipelines bolted together. This allows K3 to use screenshots as part
of active reasoning loops during coding and game development tasks, iterating
between code and live visual output rather than working blind. In addition, for
video editing, K3 edited its own teaser video from 56 source clips, handling
clip selection, motion-matched cuts, frame-accurate beat synchronization, and audio
processing through multiple rounds of revision, work that would typically take
an experienced editor one to two working days. Furthermore, in Kimi Work, K3
powers two new features, Widgets and Dashboard, which allow users to generate
interactive components directly within a chat and organize them into a
persistent, personalized view, making knowledge work more visual and continuous
rather than session-by-session.
Where K3 Stands and Why Openness Changes the Story
Kimi K3 is honest about where it sits. While K3
demonstrated frontier-level performance and consistently outperformed other
tested models, its overall performance still trails the most powerful
proprietary systems currently available. What makes it significant is not
claiming to be the best model in the world, but demonstrating that the gap
between open and closed frontier models has compressed to the point where they
are now meaningfully in the same conversation.
Open weights carry a different kind of value
than API access. A closed model, however capable, can only be used through a
vendor's interface at a vendor's price with a vendor's terms. An open model can
be studied, fine-tuned, deployed on private infrastructure, and built on by
anyone with the compute to run it. Kimi K3's full model weights are scheduled
for open release under a Modified MIT license, making it freely usable for
commercial purposes. Available now on Kimi.com, Kimi Work, Kimi Code, and the Kimi
API, with pricing at $3.00 per million input tokens and $15.00 per million
output tokens, K3 positions open-weight AI as something that no longer requires
a meaningful capability trade-off against the best available proprietary
systems.
You have exceeded your free limits for viewing our premium content
Please subscribe to have unlimited access to our innovations.
Get The Latest In Innovation
Join Our Newsletter
Where bold ideas find bold readers.
