Kimi K3 LLM hosted with 8 B300 GPUs achieves 92 tokens/sec

A user detailed their experience hosting the Kimi K3 large language model, which has 2.8 trillion parameters, using eight B300 GPUs. The setup achieved a throughput of 92 tokens per second with a time-to-first-token of approximately 1 second, at a cost of $190 per million output tokens. The user also experimented with Unsloth's Dynamic GGUF, finding it to be significantly slower and more expensive per token, though the quality was deemed acceptable. AI

IMPACT Demonstrates practical hosting costs and performance for extremely large models, informing infrastructure decisions.

RANK_REASON User-driven infrastructure deployment and performance testing of a large language model.

Read on r/LocalLLaMA →

• A100-80GB
• Kimi k3
• llama.cpp
• Modal
• Unsloth
• vLLM

• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →