Moonshot AI has developed Kimi K3, an open-weight model boasting 3 trillion parameters, making it the largest of its kind. The innovation lies not just in scale but in a novel memory management system called Kimi Delta Attention (KDA). KDA utilizes a 'Delta Rule' to update memory in place, drastically reducing computational costs associated with long context windows, unlike traditional transformer models that linearly increase memory and quadratically increase computation with context length. This approach was first tested in the smaller Kimi Linear model and is now scaled up in Kimi K3, aiming to challenge leading proprietary models. AI
IMPACT Sets a new benchmark for open-weight models and introduces a novel memory architecture that could significantly reduce training and inference costs for large context windows.
RANK_REASON Frontier-lab model release with system card. [lever_c_demoted from frontier_release: ic=2 ai=1.0]
- Claude Fable
- GPT-5.6
- Kimi K3.1
- Moonshot AI
- Delta Rule
- Kimi Delta Attention (KDA)
- Kimi K1.5
- Kimi K2
- Kimi K2.5
- Kimi K3
- Kimi Linear
AI-generated summary · Google Gemini · from 3 sources. How we write summaries →