TaDA: Calibrated Probe Gating for Task-Domain LoRA Merging
Researchers have introduced TaDA, a novel algorithm for merging task-specific and domain-specific LoRA adapters in transformer models. Unlike previous methods that applied uniform weights, TaDA leverages the observation that task adapters are stronger in shallower layers and domain adapters in deeper layers. This approach uses calibrated probe-guided gating and subspace-aware merging to create a unified LoRA adapter with no inference overhead. TaDA has demonstrated superior performance on multiple scientific QA and image classification benchmarks, outperforming existing merging techniques. AI
IMPACT This method could improve the efficiency and performance of fine-tuning large language models for specific tasks and domains.