Sparse Orthogonal Parameters Tuning for Continual Learning
Researchers have introduced SoTU, a novel method for continual learning that addresses catastrophic forgetting in pre-trained models. Unlike existing approaches that use additional adapters or prompts, SoTU focuses on merging sparse orthogonality of parameters learned from multiple tasks. This technique transforms knowledge from various domains into orthogonal delta parameters, leading to optimal feature representation for streaming data without complex classifier designs. AI
IMPACT Introduces a novel approach to continual learning that could improve model adaptability and reduce knowledge loss in sequential task learning.