Investigating Cross-Modal Skill Injection: Scenarios, Methods, and Hyperparameters
Researchers have explored a technique called cross-modal skill injection to efficiently transfer domain-specific expertise from large language models (LLMs) to vision-language models (VLMs). This method aims to induce new cross-modal capabilities without requiring extensive new training data or significant computational resources, unlike traditional fine-tuning. The study found that this skill injection is effective for instruction-following and cross-lingual tasks but less so for mathematical reasoning. Among tested methods, TA and DARE proved superior, with the research also providing a detailed analysis of their critical hyperparameter tuning. AI
IMPACT Introduces a more efficient method for adapting existing models to new domains, potentially reducing development costs and time.