A test by Runcrate revealed significant differences in the perceived speed of open-source LLMs, even when they produce answers of similar visible length. The study found that models often generate a large amount of hidden reasoning tokens before presenting the final answer, leading to much longer actual response times than benchmarks suggest. This hidden processing accounted for a substantial portion of the cost and time, with one model, Qwen3.8-Flash, taking over 14 seconds for its first user-readable token despite a fast initial stream. The research also highlighted the wide price disparity between models for equivalent outputs and noted the importance of implementing hard deadlines for streaming responses to prevent indefinite hangs. AI
IMPACT Highlights the discrepancy between benchmark speeds and real-world user experience for LLMs, impacting UI design and cost expectations.
RANK_REASON Analysis of LLM performance and cost by a third-party provider. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
