A new experimental quantization of the Kimi-K3.1-Pro-Coder model, named Kimi-K3.1-Pro-Coder (2.8T), has been developed, reducing its size from 1.77 TB to 775 GB. This was achieved through a multi-tiered quantization strategy, with different experts quantized at varying bit rates (2.81 bpw, 2.56 bpw, and 1.44 bpw) while core components remain in higher precision. The model utilizes a custom loader and offloading engine called Strata, which distributes experts across GPU VRAM, system RAM, and NVMe storage, coupled with fused Triton dequantization kernels (dr-triton) to maintain performance. AI
IMPACT Demonstrates advanced quantization and offloading techniques that could enable larger models to run on more accessible hardware.
RANK_REASON Release of a new model variant with significant size reduction and novel technical approach. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
