Kimi K3's massive scale demands intensive networking despite optimizations · 8 sources tracked

COVERAGE [8]

1. X — SemiAnalysis TIER_1 English(EN) · SemiAnalysis_ · 2026-07-18 21:35

   More efficient attention will further push context lengths from 1M to 5M+, with less context rot. Jevons’ Paradox means that making attention more efficient wil

   More efficient attention will further push context lengths from 1M to 5M+, with less context rot. Jevons’ Paradox means that making attention more efficient will lead to wider AI adoption, which will require more networking. 8/8🧵

2. X — SemiAnalysis TIER_1 English(EN) · SemiAnalysis_ · 2026-07-18 21:35

   Furthermore, with optimizations like incremental KV-cache transfers, prefill only needs to transfer the portions cached by the non-decode instance, so KV transf

   Furthermore, with optimizations like incremental KV-cache transfers, prefill only needs to transfer the portions cached by the non-decode instance, so KV transfer does not take up much networking bandwidth relative to WideEP, even before KDA. 7/8🧵

3. X — SemiAnalysis TIER_1 English(EN) · SemiAnalysis_ · 2026-07-18 21:35

   KV-cache transfer between prefill and decode happens only once per turn, while WideEP happens 120+ times per output forward pass, with potentially 500+ output t

   KV-cache transfer between prefill and decode happens only once per turn, while WideEP happens 120+ times per output forward pass, with potentially 500+ output tokens per turn. Thus, the relative KV-cache transfer savings from KDA linear attention are dwarfed by the increase in

4. X — SemiAnalysis TIER_1 English(EN) · SemiAnalysis_ · 2026-07-18 21:35

   WideEP dispatch and combine need to happen twice on every layer and for every forward pass. That means that, for each forward pass, dispatch and combine will ne

   WideEP dispatch and combine need to happen twice on every layer and for every forward pass. That means that, for each forward pass, dispatch and combine will need to run 120+ times. 5/8🧵

5. X — SemiAnalysis TIER_1 English(EN) · SemiAnalysis_ · 2026-07-18 21:35

   WideEP is a very network-intensive serving optimization that Kimi K3 will need to use. WideEP works by spreading 896 experts across many GPUs within a scale-up

   WideEP is a very network-intensive serving optimization that Kimi K3 will need to use. WideEP works by spreading 896 experts across many GPUs within a scale-up domain, then dispatching tokens to the correct GPU and combining the results. 4/8🧵 https://t.co/ZciwDfWfZR

6. X — SemiAnalysis TIER_1 English(EN) · SemiAnalysis_ · 2026-07-18 21:35

   Because Kimi K3 has 2.8 trillion parameters, even at MXFP4, each forward pass will require 1.5 TB of HBM bandwidth. This means that, even with spec decode, serv

   Because Kimi K3 has 2.8 trillion parameters, even at MXFP4, each forward pass will require 1.5 TB of HBM bandwidth. This means that, even with spec decode, serving it profitably at a reasonable level of interactivity requires aggregating many chips together over a high-bandwidth

7. X — SemiAnalysis TIER_1 English(EN) · SemiAnalysis_ · 2026-07-18 21:35

   While it is true that Kimi K3 uses Kimi Delta Linear Attention (KDA) in 3 out of every 4 layers and that KDA reduces KV-cache transfer bandwidth by up to 10x co

   While it is true that Kimi K3 uses Kimi Delta Linear Attention (KDA) in 3 out of every 4 layers and that KDA reduces KV-cache transfer bandwidth by up to 10x compared with comparable full global-attention models, the important missing piece is that Kimi K3 requires WideEP to http…

8. X — SemiAnalysis TIER_1 English(EN) · SemiAnalysis_ · 2026-07-18 21:35

   Similar to DeepSeek in January 2025, Panicans may think that the AI networking switch TAM will massively shrink because Kimi K3 uses KDA Attention, which reduce

   Similar to DeepSeek in January 2025, Panicans may think that the AI networking switch TAM will massively shrink because Kimi K3 uses KDA Attention, which reduces KV-transfer networking bandwidth by up to 10x. But the opposite is true, as we explain below. 👇️ 1/8🧵 https://t.co/FNr…