From Observation to Intervention: A Causal Audit of Expert Importance in Mixture-of-Experts Models
A new research paper investigates the effectiveness of interpretability methods in Mixture-of-Experts (MoE) models. The study found that common metrics used to predict which experts can be removed without impacting performance do not reliably correlate with causal expert importance. Across three different MoE architectures, observational data failed to predict expert dispensability, suggesting current pruning techniques may succeed due to redundancy rather than precise identification of critical components. AI
IMPACT Challenges current assumptions in MoE model interpretability and pruning, potentially leading to more robust methods.