Correct When Paired, Wrong When Split: Decoupling and Editing Modality-Specific Neurons in MLLMs
Researchers have identified a significant issue in knowledge editing for Multimodal Large Language Models (MLLMs), termed 'editing decoupling failure.' This occurs when updates to an MLLM's knowledge are effective with paired multimodal inputs (like text and images) but revert to outdated information when presented with unimodal inputs (text only or image only). The problem stems from knowledge being distributed across modality-specific pathways rather than unified. To address this, a new method called DECODE has been proposed, which aims to disentangle and localize these modality-specific neuron groups for more targeted and effective knowledge updates across different input types. AI
IMPACT This research could lead to more robust and reliable knowledge updates in multimodal AI systems, improving their accuracy and consistency across different input formats.