研究人员正在为混合专家（MoE）模型开发先进的路由机制，特别是那些使用低秩适应（LoRA）的模型。新的方法如VI-MoLE和CARE不再仅仅基于不确定性进行路由，而是专注于根据“信息价值”或“置信度自适应路由”来分配计算资源。这些方法旨在优化专家激活，以降低风险并提高准确性，尤其是在分布变化的情况下。论文还探讨了MoE路由中专家重叠和依赖性控制的潜在几何原理，表明虽然专家可能共享表示空间，但它们的协调使用对于性能至关重要。
AI
arXiv:2608.02528v1 Announce Type: new Abstract: Mixtures of low-rank adaptation experts increase parameter-efficient capacity by routing each input through a subset of adapters. Recent dynamic routers activate more experts when the router or prediction is uncertain. This rule sil…
arXiv:2607.26052v2 Announce Type: replace Abstract: Mixture-of-Experts (MoE) variants of Low-Rank Adaptation (LoRA) route every token to a fixed number of experts $k$. Tokens differ in how uncertain the model is about them, so a single k over-spends on easy tokens and under-serve…
arXiv:2607.28670v1 Announce Type: new Abstract: A stochastic Gumbel-Top-$K$ router defines, for every token of a mixture-of-experts (MoE) model, a \emph{routing law}: a distribution over ordered expert lists and mixture weights. We ask which \emph{joint} distributions over the ro…
arXiv:2607.28308v1 Announce Type: new Abstract: Sparse mixture-of-experts (MoE) language models route each token to multiple experts, suggesting a geometric account of their benefit: co-selected experts should contribute distinct representation directions. Existing evidence often…
Sparse mixture-of-experts (MoE) language models route each token to multiple experts, suggesting a geometric account of their benefit: co-selected experts should contribute distinct representation directions. Existing evidence often conflates route coherence, candidate quality, a…
Sparse mixture-of-experts (MoE) language models route each token to multiple experts, suggesting a geometric account of their benefit: co-selected experts should contribute distinct representation directions. Existing evidence often conflates route coherence, candidate quality, a…
arXiv cs.LG
TIER_1English(EN)·Haomiao Sun, Fang He, Congyuan Ji, Xindi Tang·
arXiv:2607.23467v1 Announce Type: new Abstract: We study an integrated pickup-and-delivery problem on sparse, non-Euclidean networks that jointly optimizes cyclic routing, cargo flow allocation, and cross-cycle service. The tight coupling of these operational constraints creates …
<p><em>Originally published at <a href="https://getyourdozai.blogspot.com/2026/08/mixture-of-experts-moe-explained-how.html" rel="noopener noreferrer">https://getyourdozai.blogspot.com/2026/08/mixture-of-experts-moe-explained-how.html</a></em></p> <p>Mixture of Experts (MoE) is a…
