Tencent has released Hy3, a 295-billion-parameter Mixture-of-Experts (MoE) model that utilizes only 21 billion active parameters per forward pass, significantly reducing inference costs. This MoE architecture, featuring 80 layers with 192 routed experts per layer, allows for selective computation, making its operational cost comparable to a 21B dense model. Additionally, Hy3 incorporates a Multi-Token Prediction layer that speeds up token generation and is compatible with existing serving frameworks like vLLM and SGLang, while also offering different reasoning modes and tool-calling capabilities. In parallel, VIDRAFT has demonstrated Ourbox-35B-JGOS, a 34.7B parameter MoE model that can run on a CPU due to its sparse nature, where only approximately 3 billion parameters are active per token. This efficiency is attributed to memory bandwidth optimization, streaming only active experts rather than all weights, which drastically reduces memory traffic compared to dense models. Ourbox-35B-JGOS achieves a competitive score of 86.4% on GPQA Diamond and its performance can be reproduced using public GGUF weights and demos. AI
IMPACT Sparse MoE architectures enable more efficient model deployment, potentially lowering hardware requirements and increasing accessibility for advanced AI capabilities.
RANK_REASON The cluster describes new model architectures and their performance characteristics, fitting the research category.
- GGUF
- GPQA Diamond
- Nvidia B200
- Ourbox-35B-JGOS
- Q3_K_M
- VIDRAFT
- VKUE
- Apache Software License 2.0
- H200
- mixture of experts
- Multi Token Prediction
- SGLang
- Tencent
- vLLM
AI-generated summary · Google Gemini · from 2 sources. How we write summaries →
