Local LLM inference with 96GB VRAM fails to beat paid APIs on cost

A user detailed their two-week effort to optimize a local LLM setup with 96GB of VRAM across four RTX 3090 GPUs, aiming to replace paid cloud APIs. Despite achieving approximately 105 tokens/second and implementing optimizations like increased batch size and KV cache quantization, the system's CPU orchestration bottleneck resulted in only 6% GPU utilization. Ultimately, the high power consumption and hardware depreciation made the local setup economically unviable for interactive work compared to paid APIs, though it remains suitable for privacy-focused or batch tasks. AI

IMPACT Highlights the economic challenges of running large local LLMs for interactive tasks compared to cloud APIs.

RANK_REASON User-generated content detailing personal experience and technical findings.

Read on dev.to — LLM tag →

• GLM 5.2
• llama.cpp
• OpenWebUI
• Qwen3.6-35B-A3B
• Qwen3-Coder-Next
• RTX 3090

• infra
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →