New theory explains why Zeroth-Order adaptation reduces model forgetting

Researchers have developed a new theoretical framework, Randomized Shaping Theory, to explain why Zeroth-Order (ZO) adaptation methods in continual learning may lead to less forgetting than first-order (FO) methods. The theory suggests that ZO adaptation, when properly analyzed, can preserve more previously acquired knowledge by selectively contracting anisotropic components of adaptation. This theoretical insight has led to a new algorithm called RISE, which applies calibrated ZO shaping to exact FO gradients within parameter blocks to improve the stability-plasticity tradeoff in continual learning. AI

影响 Introduces a theoretical explanation for improved continual learning, potentially leading to more robust AI systems that retain knowledge over time.

排序理由 The cluster contains a new academic paper detailing a theoretical framework and a proposed algorithm for continual learning. [lever_c_demoted from research: ic=1 ai=1.0]

在 arXiv cs.LG 阅读 →

• Continual learning
• First-Order adaptation
• Randomized Shaping Theory
• RISE
• Zeroth-Order adaptation

• 论文
• 安全

AI 生成摘要 · Google Gemini · 来自 1 个来源。 我们如何撰写摘要 →