32B LLM on a 2008 Xeon: When RAM Matters More Than VRAM
An experiment explored running a 32-billion parameter LLM on a 2008-era server with 64GB of RAM but no dedicated GPU, contrasting it with a modern laptop with a GeForce RTX 4070. Despite the older hardware's significantly slower inference speed (0.01 tokens/sec), it successfully ran the model entirely in system RAM, a feat the modern laptop struggled with due to insufficient combined VRAM and RAM. The experiment also highlighted that even large models may not perform well on specialized programming tasks like generating Forth code without specific training. AI
IMPACT Demonstrates that sufficient system RAM can enable LLM execution where VRAM is a bottleneck, albeit with significant speed trade-offs.