A Reddit user has optimized inference for the Qwen3.8-27B model on an RTX 3090 graphics card, achieving up to 82 tokens per second for single requests and a peak of 672 tokens per second with concurrent requests. The optimizations involve W4A16 quantization, fp8 KV cache, and int8 for lm_head and embed_tokens, reducing VRAM usage to 14.2 GB while supporting a context length of up to 195k tokens. The user has shared a GitHub repository with the necessary patches and setup instructions, noting it runs via vLLM and has been tested on Linux. AI
IMPACT Demonstrates advanced optimization techniques for running large language models on consumer-grade hardware.
RANK_REASON User-driven optimization of an existing model on consumer hardware.
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
