Brief

GLM-4, a bilingual Chinese-English model developed by Tsinghua University and Zhipu AI, is highlighted for its strong performance in handling both languages natively. Optimized for agent workflows and featuring a Mixture of Experts architecture, it offers efficient inference and a long context window of up to 128K tokens. This model is particularly beneficial for developers building tools that require seamless integration of Chinese and English content, unlike many English-centric open-source alternatives. AI

Researchers have developed ZipMoE, a system designed to make Mixture-of-Experts (MoE) large language models more efficient for on-device deployment. ZipMoE utilizes lossless compression and a cache-affinity scheduling approach to reduce memory footprint and improve inference speed without sacrificing model accuracy. Experiments show significant reductions in latency and increases in throughput on edge devices, shifting the inference bottleneck from I/O to computation. AI

Fireworks AI has identified critical numerical parity bugs that can arise when training and serving large language models, particularly Mixture-of-Experts (MoE) architectures. These discrepancies, stemming from the non-associative nature of floating-point arithmetic and differing summation orders in distributed training versus inference, can lead to subtle but significant issues. Such drift can compromise the integrity of reinforcement learning from human feedback (RLHF) due to altered log probabilities and erode customer trust in fine-tuned models. AI

Alibaba's Qwen3-Coder-Next, an 80 billion parameter model with 3 billion active parameters, has achieved a 70.6 score on the SWE-Bench Verified benchmark. This performance is notable as it rivals top closed-source models while offering downloadable weights under the Apache 2.0 license. The model employs a sparse Mixture-of-Experts architecture and a hybrid attention mechanism, combining linear attention for long contexts with standard attention for global context reconstruction. AI

The human brain's extreme energy efficiency, estimated to be 10,000 times greater than current AI models, is attributed to its sparse and localized processing. While techniques like mixture-of-experts offer a path toward similar efficiency in AI by using specialized sub-networks, they may reduce the benefits of superposition. Superposition, a dense shared representational space, allows neural networks to compress multiple features into the same neurons, contributing to their power but hindering interpretability. The author posits that more segmented architectures could weaken superposition, potentially making AI models easier to inspect and govern, and seeks a balance between efficiency, power, and interpretability. AI

A new research paper explores the trade-offs between performance, energy consumption, and privacy when running large language models on mobile devices. The study developed an experimental pipeline to measure these factors on an Android device, testing eight LLMs. Findings indicate that model architecture, rather than quantization, is key for energy efficiency, with Mixture-of-Experts models showing promise for balancing storage and power usage. AI

Researchers have developed a new framework called HDMoE to improve multimodal cancer survival prediction. This hierarchical decoupling-fusion mixture-of-experts approach aims to better integrate data from sources like whole slide images and genomic profiles. The framework addresses limitations in existing methods by reducing redundant information before feature decoupling and by modeling fine-grained relationships within and between modalities. AI

Researchers have introduced FedCoE, a novel framework for Federated Learning that aims to balance global generalization with local personalization. Unlike traditional methods that struggle with non-IID data or overfit to local information, FedCoE utilizes a dual-level Mixture-of-Experts approach. This system maintains independent global expert models and uses a shared gating network to manage client-expert correlations, preventing expert drift. FedCoE also includes an adaptive mechanism to help new clients quickly utilize global experts without extensive local training, showing significant accuracy improvements in both general and cold-start scenarios. AI

Researchers have developed FPED, a novel Mixture-of-Experts (MoE) framework designed for interpretable brain decoding using fMRI data. This approach explicitly models different functional brain networks as specialized experts, utilizing adaptive routing to capture their combined contributions to visual semantic understanding. FPED aims to overcome limitations of current methods that flatten fMRI signals, thereby disrupting the brain's natural network topology and reducing neuroscientific interpretability. The framework demonstrates competitive performance with a small parameter count and offers transparent insights into the correspondence between brain networks and semantic processing. AI

Researchers have introduced SpikingMoE, a novel framework that combines Spiking Neural Networks (SNNs) with a Mixture-of-Experts (MoE) architecture. This approach utilizes a spike-driven prompt (SDprompt) for biologically plausible, input-dependent routing of information to different expert modules. Designed for neuromorphic hardware, SpikingMoE aims to enhance energy efficiency in visual recognition tasks while maintaining competitive performance, achieving high accuracy on CIFAR-10 and CIFAR-100 datasets. AI