87GB LLM runs on CPU via quantization; vecq tool targets on-device embeddings

A recent benchmark demonstrated that the Qwen3.8-Flash-Next large language model, with a file size of 87.2 GiB, can run on a CPU at a speed of 8.34 tokens per second using the llama.cpp framework. This performance is achieved through 4-bit quantization, which significantly reduces the model's size. The benchmark also revealed that at a context length of 245K, the performance advantage of 96GB of VRAM over 24GB diminishes, and placing the model's embedding table on the GPU drastically improves decode speed. To address similar size constraints for on-device applications, the vecq tool was developed, offering training-free vector quantization that reduces embedding size by up to 4.78x with high recall. AI

IMPACT Demonstrates significant advancements in model quantization and embedding techniques, enabling larger models to run on less powerful hardware and paving the way for on-device AI applications.

RANK_REASON Benchmark and technical discussion of LLM quantization and embedding optimization. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

• CPU
• EmbeddingGemma
• llama.cpp
• Qwen3.8-Flash-Next
• r/LocalLLaMA
• Rust
• VRAM

• infra
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →