When Shared Knowledge Hurts: Spectral Over-Accumulation in Model Merging
Researchers have introduced Singular Value Calibration (SVC), a novel post-processing technique designed to improve model merging by addressing the issue of spectral over-accumulation. This method quantifies and rescales overlapping spectral directions in shared knowledge across tasks, preventing inflated singular values and subspace bias. SVC, which is training-free and data-free, has demonstrated consistent performance improvements on vision and language benchmarks, enhancing existing merging baselines and achieving state-of-the-art results. AI
IMPACT Improves model merging techniques, potentially leading to more efficient and effective deployment of specialized AI models.