CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning
Two new research papers propose novel approaches to continual learning in large language and vision-language models, aiming to mitigate catastrophic forgetting. CP-MoE introduces a transient expert to guide updates and preserve knowledge, while MoRAM utilizes fine-grained rank-1 adapters as memory units to enable content-addressable retrieval. Both methods demonstrate improved performance on benchmarks, offering better trade-offs between plasticity and stability compared to existing Mixture-of-Experts techniques. AI
IMPACT These papers introduce novel techniques for continual learning, potentially improving the ability of large models to adapt to new information without forgetting previous knowledge.