180B MoE model runs on 32GB RAM laptop, SSD is the bottleneck

A 180-billion parameter Mixture of Experts (MoE) model, Darwin-180B-RSI, was successfully run on a laptop with 32GB of RAM, achieving approximately 3 tokens per second using only the CPU. Smaller MoE models, POCKET-26B and POCKET-35B, which fit entirely into memory, performed significantly faster at over 11 tokens per second on the same hardware. The primary bottleneck for the larger model was identified as the Solid State Drive (SSD) read speed, with each token generation requiring substantial data to be re-read from the SSD due to limited cache residency. AI

IMPACT Demonstrates that large MoE models can be run on consumer hardware, highlighting SSD performance as a key factor for efficient inference.

RANK_REASON The item details a technical experiment and performance analysis of running large language models on consumer hardware, focusing on hardware bottlenecks. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

• Arm Holdings
• central processing unit
• Darwin-180B-RSI
• GGUF
• graphics processing unit
• llama.cpp
• POCKET-26B
• POCKET-35B
• Q4

• infra
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →