Benchmarking vLLM on a single GeForce RTX 3060 revealed significant performance variations based on configuration settings. A single parameter, max-num-seqs, demonstrated a 19x increase in time-to-first-token (TTFT) when increased from 32 to 128 under stress conditions. The study also found that the llm-jp-1.8B model served requests 3.6 times faster than Swallow-8B AWQ, attributed partly to vLLM's optimized kernels. Additionally, a reasoning model using Swallow-8B AWQ exhibited non-deterministic output even at temperature 0, with 10 out of 50 outputs differing between runs. AI
IMPACT Highlights the critical need for stress testing configurations and verifying model determinism for reliable LLM deployments.
RANK_REASON Benchmarking results and analysis of LLM serving performance. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
