A user has achieved impressive performance with the Qwen3.8-27B model on a single RTX 5090 GPU, reaching 880 tokens/second with 4-bit NVFP4 quantization and a full 262k context. This speed was attained using the NInfer engine, which the user reports as significantly faster than other engines like llama.cpp. The benchmark results indicate that this NVFP4 quantized model performs comparably to integer quantized versions on tasks like HumanEval+ and AIME, while being substantially faster. AI
IMPACT Demonstrates significant speedups for local LLM inference on consumer hardware, potentially lowering barriers to entry for advanced model usage.
RANK_REASON User-reported performance benchmark of a specific model on consumer hardware using a novel engine.
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
