A new CUDA megakernel has been developed for the Qwen 3.8 27B model, offering significantly faster inference speeds compared to llama.cpp on a single RTX 3090 GPU. Benchmarks indicate that the megakernel achieves 140 tokens/sec in code writing tasks, nearly double the speed of llama.cpp, with comparable accuracy and minimal KL divergence. The project has also seen improvements in model loading and quantization support, with further contributions merging into the codebase. AI
IMPACT This development offers a significant speed-up for running the Qwen 3.8 27B model locally, potentially enabling more complex applications on consumer hardware.
RANK_REASON New optimized kernel for an existing open-source model. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
