llama.cpp adds support for Tencent's Hy3 model with MTP speculative decoding

The llama.cpp project has integrated support for Tencent's Hy3 model, also known as hy_v3. This update includes the implementation of MTP speculative decoding, which allows for multi-token prediction. The Hy3 model is a large 299 billion parameter Mixture-of-Experts model featuring 80 layers plus an additional MTP layer. AI

IMPACT Enhances local LLM inference capabilities by adding support for a large MoE model.

RANK_REASON This is a pull request to add support for a specific model to an existing open-source project, not a new model release from a frontier lab.

Read on r/LocalLLaMA →

• ggml-org
• hy_v3
• llama.cpp
• MTP Speculative Decoding
• satindergrewal
• tencent/Hy3

• model release
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →