A comparison of three inference engines—basalt, cachy-flash, and ninfer-ext—for the Qwen3.8-FN model on an NVIDIA RTX 5090 GPU revealed that basalt generally offered superior performance. Across various context lengths, basalt consistently achieved higher tokens-per-second rates for both prefill and decoding operations. While cachy-flash showed competitive prefill speeds at shorter contexts, basalt maintained its lead as context length increased, and ninfer-ext lagged behind both. AI
IMPACT Optimizing inference engines can significantly improve the speed and efficiency of running large language models locally.
RANK_REASON Benchmark comparison of inference engines for a specific model. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
