Budget LLM Benchmarking: Consumer GPUs Linked via RPC

A personal project benchmarks six open-source LLMs on a two-machine cluster utilizing consumer-grade GPUs and RPC for model layer distribution. The setup aims to provide a cost-effective solution for users with limited budgets who cannot afford professional-grade hardware. The author details the hardware configuration, including network bandwidth and VRAM limitations, and discusses challenges encountered with tensor-split ordering and model architecture differences between dense models like Llama 3.3 and MoE models like Qwen3.6. AI

IMPACT Demonstrates a practical, budget-friendly approach to running large LLMs on consumer hardware, potentially enabling wider experimentation.

RANK_REASON The item details a personal experiment benchmarking open LLMs on custom hardware, including methodology and results. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

• Claude 3.5 Sonnet
• llama
• Llama 3.3 70B Instruct
• llama.cpp
• open LLMs
• Opus 4.8
• Qwen3.6
• Qwen3.6 35B-A3B
• RPC Cluster
• RTX 5090

• infra
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →