Brief

Meta's Llama 4 Scout, a 109 billion parameter mixture-of-experts model, requires approximately 25GB of VRAM for usable performance at Q4_K_M quantization. The RTX 5090 with 32GB of VRAM is presented as the sole single consumer GPU capable of running the model locally. For a more cost-effective local solution, a dual RTX 3090 setup offers comparable performance and more VRAM for a similar price, though it involves greater complexity. Cloud GPU instances are recommended for users who only need to run the model occasionally. AI

Running large open-source language models locally can lead to out-of-memory errors, even if the model's weights seem to fit within the available VRAM. This is primarily due to the significant memory required for the KV cache, which scales with context length, and intermediate activation memory during inference. Developers can address these issues by profiling memory usage with tools like PyTorch's memory snapshot, applying appropriate quantization techniques to model weights and the KV cache, and managing memory fragmentation. AI

The NVIDIA RTX 5090, released in early 2025, offers a significant upgrade for local LLM users with its 32GB of GDDR7 memory, compared to the RTX 4090's 24GB of GDDR6X. This increased VRAM allows the 5090 to comfortably run larger models, such as 34B parameter models at higher quantization levels, and even 70B models at lower quantizations, which are impossible on the 4090. While the 5090 comes at a higher price point of approximately $2,000, it provides substantial benefits for those needing to run larger models or requiring more VRAM for longer context windows, whereas the RTX 4090 remains a strong option for users primarily working with smaller models. AI

This series of guides provides comprehensive instructions for setting up and running large language models (LLMs) locally on Linux systems. It details hardware and software prerequisites, recommends using llama.cpp for its balance of performance and ease of use, and covers model selection, quantization, and API integration. The guides also include steps for setting up systemd services for 24/7 operation, monitoring performance, and optimizing for various hardware constraints. AI

Researchers have developed ChunkFT, a new framework designed to make full-parameter fine-tuning of large language models more memory-efficient. This method allows for gradient computation on dynamic subsets of model parameters, reducing the need for extensive GPU memory. Experiments with Llama 3 models demonstrated significant memory savings, enabling fine-tuning on consumer-grade hardware, and achieved performance comparable to or exceeding traditional full fine-tuning methods on various downstream tasks. AI