PulseAugur
EN
LIVE 08:25:43

763B MoE model runs on dual 96GB GPUs via expert offload

By PulseAugur Editorial · [1 sources] ·

A technical guide details how to run the 763 billion parameter DeepSeek-V4.1-Flash model on consumer-grade hardware, specifically two Nvidia RTX PRO 6000 Blackwell GPUs with 96GB of VRAM each. The solution involves offloading idle "experts" of the Mixture of Experts (MoE) architecture to the server's host RAM, a technique that successfully reduced the VRAM requirement per GPU from over 100GB to under 73GB. This method, implemented via a custom stack involving ExLlamaV3 and vLLM, allows for efficient inference, with measurements showing a capacity of four concurrent streams at approximately 40 tokens per second each, even with extensive context lengths. AI

IMPACT Enables running large MoE models on more accessible hardware, potentially lowering the barrier for advanced AI experimentation.

RANK_REASON This item details a technical method for running a large model on consumer hardware, not a new model release or research breakthrough.

Read on dev.to — LLM tag →

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →

763B MoE model runs on dual 96GB GPUs via expert offload

How we ranked this

Signal score
22 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Tool
This item details a technical method for running a large model on consumer hardware, not a new model release or research breakthrough.
Source corroboration
Single-source cluster
Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics
infra, model release
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
Breaking (< 6h)
Fresh story with cross-source coverage still developing. Ranking may shift as more sources report.

Full methodology in our editorial standards.

COVERAGE [1]

  1. dev.to — LLM tag TIER_1 English(EN) · InferenceKey ·

    A 763B MoE on two 96 GB GPUs: put the idle experts in host RAM, then measure per stream

    <p>The model has 763B parameters, the server has two graphics cards, and the gap was about six gigabytes per card. DeepSeek-V4.1-Flash, even compressed to 3 bits, wanted 102.1 GiB of VRAM (the memory on the GPU itself) on each of two RTX PRO 6000 Blackwell cards, and each card of…

RELATED ENTITIES

RELATED TOPICS