Few-shot learning evaluation protocols may overestimate model performance

A new paper critically examines the assumptions behind few-shot learning evaluations, particularly the common practice of pre-training models on large auxiliary datasets. Researchers found that pre-training, even with classes disjoint from the target episodes but within the same visual domain, significantly overestimates performance. The study suggests that out-of-domain pre-training is more realistic for scarce target-domain data, and surprisingly, label-free augmentation strategies can achieve comparable results to supervised out-of-domain pre-training. The paper advocates for moving away from in-domain pre-training as the default evaluation protocol to better reflect real-world scenarios. AI

IMPACT Challenges standard evaluation methods for few-shot learning, potentially leading to more realistic performance assessments and influencing future model development.

RANK_REASON Academic paper published on arXiv detailing a critical examination of a machine learning evaluation protocol. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• Alexandro Galan-Cuenca
• arXiv
• Computer vision and pattern recognition
• Few-shot learning
• pre-training

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →