A user has detailed their benchmark results for running the DeepSeek V4.1 Flash model locally on a personal Linux machine. The setup achieved a generation speed of 42 tokens per second with a 256K context window and vision input enabled, utilizing three GPUs. The user emphasizes that these results are from a custom-quantized build and a patched engine, making them not directly comparable to stock configurations of tools like llama.cpp or Ollama. The performance is attributed to a strategic placement of model components across GPUs, system RAM, and an Optane SSD to manage the MoE architecture and large context window. AI
IMPACT Demonstrates high performance for large context models on consumer hardware, potentially lowering barriers for local AI deployment.
RANK_REASON User-generated benchmark of an open-source model on custom hardware. [lever_c_demoted from research: ic=1 ai=1.0]
- DeepSeek V4.1 Flash
- EXL3
- Linux
- llama.cpp
- Muse
- Nvidia GeForce RTX 5070 Ti
- Ollama
- Optane SSD
- RTX 5060 Ti
- vLLM
- Zenn
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →