Run 100B+ MoE LLMs on a Single RTX 4090 via Expert Offloading

A technical guide explains how to run large Mixture of Experts (MoE) Large Language Models (LLMs) on a single consumer-grade GPU, specifically an RTX 4090 with 24GB of VRAM. The method leverages the MoE architecture, where only a fraction of the model's parameters are active for each token, allowing the remaining "expert" layers to be offloaded to system RAM and processed by the CPU. This technique significantly reduces the VRAM requirement but introduces a bottleneck in system RAM bandwidth, impacting inference speed. The guide provides a script to estimate memory usage and performance before downloading large models, suggesting that models around 30-50 billion parameters are more feasible for systems with 64GB of RAM, rather than the 125B parameter models often benchmarked. AI

IMPACT Enables running larger LLMs on consumer hardware by optimizing VRAM usage, potentially lowering the barrier to entry for local AI experimentation.

RANK_REASON Guide on using existing software (llama.cpp) to run models on specific hardware.

Read on dev.to — LLM tag →

• DDR5 SDRAM
• Hacker News
• Llama 2
• Llama 3
• llama.cpp
• LLMs
• Mixture of Experts (MoE)
• RTX 4090

• infra
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →