Information-Theoretic Requirements for Gradient-Based Task Affinity Estimation in Multi-Task Learning
A new research paper published on arXiv introduces a principled framework for understanding multi-task learning outcomes. The study identifies a critical requirement for gradient-based task affinity estimation: tasks must share training instances for gradient conflicts to accurately reveal relationships. Below 30% sample overlap, gradient correlations become indistinguishable from noise, while above 40%, they reliably recover known biological structure. This finding offers a potential explanation for the inconsistent results observed in multi-task learning over the past seven years, as many standard benchmarks fall below the meaningful threshold. AI
IMPACT Identifies a fundamental requirement for improving multi-task learning performance and reliability.