Qwen3.5 35B model runs 55 tok/s on RTX 5060 Ti with float8 optimization

A user on Reddit has shared an optimization for running the Qwen3.5 35B model using float8 precision, achieving speeds of 55 tokens per second on an RTX 5060 Ti. This performance significantly surpasses that of llama.cpp running the same model at Q8 quantization. The user also noted that further speed improvements are possible with techniques like MTP (multiple tokens produced), and is planning to document the optimization method in a blog post. AI

IMPACT Demonstrates potential for faster local inference on consumer GPUs, enabling broader access to larger models.

RANK_REASON User-driven optimization for running a specific LLM on consumer hardware.

Read on r/LocalLLaMA →

• llama.cpp
• Qwen3.5 35B
• RTX 5060 Ti

• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →