I compared all specs of the major GPUs/machines that are being used here, because bandwidth is not everything. Some of ya'll need a reality check.
A Reddit user on r/LocalLLaMA has analyzed various GPUs and machines for their suitability in running large language models, emphasizing the importance of prefill performance over raw generation speed. The analysis suggests that while some high-end GPUs like the 3090 might be overkill for single-stream use, older cards like the P100 offer significant value for their memory and bandwidth. The user also noted that Mac Studio is overpriced and inefficient compared to other options, and is seeking user-submitted power data to further refine their performance charts. AI
IMPACT Provides insights into hardware choices for AI operators running local LLMs, focusing on performance trade-offs.