Olmoe
PulseAugur coverage of Olmoe — every cluster mentioning Olmoe across labs, papers, and developer communities, ranked by signal.
3 day(s) with sentiment data
-
New BadMoE attack targets Mixture-of-Experts LLMs via routing manipulation
Researchers have developed a new method called BadMoE to exploit vulnerabilities in Mixture-of-Experts (MoE) large language models. This technique targets the routing mechanism of MoE architectures, which selectively ac…
-
New research explores expert deactivation in LLMs, revealing performance impacts
Researchers have investigated methods for deactivating experts in large language models during inference, finding that the approach significantly impacts performance. Their study compared four deactivation techniques—re…
-
New SafeStart method restores safety in faster MoE LLM serving
Researchers have developed SafeStart, a new method to mitigate safety risks associated with optimizing Mixture-of-Experts (MoE) language models for faster serving. While reducing the number of experts used per token can…
-
New research explores MoE expert pruning, introduces 'tilted novice' operator
A new research paper explores training-free expert pruning techniques for Mixture-of-Experts (MoE) models, focusing on how to handle pruned experts and the routing semantics. The study compares various replacement opera…
-
MoE language models show limited stereotype control via routing
A new research paper explores methods for controlling stereotypes in Mixture-of-Experts (MoE) language models by adjusting routing mechanisms. The study introduces the Fairness-Aware Routing Equilibrium (FARE) framework…
-
Hugging Face releases Olmo-core 3 for scalable trillion-parameter MoE training
Hugging Face has released Olmo-core 3, an open-source framework designed to scale the training of Mixture-of-Experts (MoE) large language models to trillion-parameter sizes. This new system addresses the computational i…
-
New research reveals predictive structure in sparse MoE routing
Researchers have identified residual predictive structure in the routing mechanisms of sparse mixture-of-experts (MoE) models. By analyzing frozen OLMoE and JetMoE models, they found that incorporating expert selections…
-
New RAPTOR framework enhances private training for MoE AI models
Researchers have developed RAPTOR, a novel framework for differentially private training of Mixture-of-Experts (MoE) models. Existing methods treat these sparse models as dense blocks, leading to issues like gradient su…
-
New research explores advanced routing techniques for Mixture-of-Experts models · 4 sources tracked
Researchers are exploring new methods to improve the performance and specialization of Mixture-of-Experts (MoE) models. One approach focuses on aligning the geometric structures of routing states across different layers…
-
New pruning method for LLMs uses information boundaries beyond compression scores
Researchers have developed a new method for pruning large language models (LLMs) that goes beyond traditional compression scores. This approach, termed "Information Boundaries," aims to identify the most effective param…
-
New LLM Pruning Method Prioritizes Worst-Case Group Performance
Researchers have developed a new method for evaluating and pruning Large Language Models (LLMs) that focuses on group-robustness, addressing scenarios where standard compression scores might select suboptimal models. Th…
-
New MoE routing methods optimize expert use beyond simple uncertainty
Researchers are developing advanced routing mechanisms for Mixture-of-Experts (MoE) models, particularly those using Low-Rank Adaptation (LoRA). Instead of simply routing based on uncertainty, new methods like VI-MoLE a…
-
New EPnG framework enhances MoE model fine-tuning efficiency
Researchers have developed EPnG, a novel framework for parameter-efficient fine-tuning of Mixture-of-Experts (MoE) models. This method adaptively reallocates fine-tuning capacity by pruning under-utilized experts and gr…
-
New research explores efficient Mixture-of-Experts models
Researchers have proposed several novel approaches to enhance the efficiency and capabilities of Mixture-of-Experts (MoE) language models. One method, "Expert Tying," reduces memory footprint by sharing expert parameter…
-
Study: Language model circuits vary by architecture
A new study published on arXiv investigates how different language model architectures implement similar task functionalities. Researchers found that the specific circuits responsible for task execution vary significant…
-
New framework enhances MoE LLMs on noisy analog hardware
Researchers have introduced ROMER, a post-training calibration framework designed to enhance the robustness of Mixture-of-Experts (MoE) Large Language Models (LLMs) when deployed on analog Compute-in-Memory (CIM) system…
-
Apple researchers unveil SpecMD for faster MoE model inference
Apple's machine learning research team has published a paper detailing SpecMD, a new framework for evaluating Mixture-of-Experts (MoE) model caching policies. Their experiments show that traditional caching assumptions …