GPU performance for LLMs: Community data sought to reveal real-world token/sec rates

A blog post on dev.to is seeking community input to determine the actual performance of GPUs when running large language models. The author explains that while spec sheets provide theoretical bandwidth, the real-world performance, or "eta," is often lower due to factors like memory controller limitations and kernel inefficiencies. The post asks readers to run a specific command using llama.cpp and share their GPU model, the model and quantization used, and the resulting tokens per second to help build a table of actual performance metrics. AI

IMPACT Provides a practical method for users to benchmark their hardware and contributes to a better understanding of real-world LLM inference performance.

RANK_REASON Blog post seeking community data to analyze GPU performance for LLMs.

Read on dev.to — LLM tag →

• 3090
• 4080
• 4090
• A100 80GB
• CUDA
• GGUF
• GPU
• H100 SXM
• Llama-3.1:8b
• llama.cpp
• LLM
• Q4_K_M
• tok/s

• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →