llama.cpp MTP modified for lower VRAM, faster Qwen model inference

A modified version of llama.cpp's Multi Token Prediction (MTP) has been developed to reduce memory usage, allowing larger models like Qwen 27B to run on systems with limited VRAM, such as 16GB. This new 'Compact Rollback MTP' feature allows users to specify how many rollback states the model keeps in VRAM, enabling higher token generation limits with less context cost. The modification also includes adaptive speculative decoding, which dynamically adjusts the MTP draft limit based on recent draft acceptance, further optimizing performance. AI

IMPACT Enables running larger language models on consumer hardware, potentially increasing accessibility and use cases for AI applications.

RANK_REASON This is a modification of an existing tool (llama.cpp) to improve performance for specific models (Qwen) on limited hardware, rather than a new model release or core research.

Read on r/LocalLLaMA →

• Compact Rollback MTP
• llama.cpp
• Multi Token Prediction
• Qwen

• infra
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →