Huawei has open-sourced its new Mixture-of-Experts (MoE) model, openPangu-2.0-Pro. This model boasts a total of 505 billion parameters with 18 billion activated, and supports a context length of 512,000 tokens. It was trained on 34 trillion tokens and incorporates advanced training techniques including unified SFT for slow and fast thinking, multiple specialist RL training, and on-policy distillation. AI
IMPACT This open-source release provides researchers and developers with a powerful new MoE model, potentially accelerating advancements in efficient large language model development and deployment.
RANK_REASON Open-source release of a large language model from a major tech company. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
