A new paper highlights significant flaws in how automated machine learning (AutoML) systems are compared, particularly under short time budgets. The study reveals that a system called Orcetra appeared to outperform competitors like FLAML and AutoGluon due to protocol defects, including testing on the actual test split and failing to enforce time limits. When these issues were corrected by using a validation split and enforcing external deadlines, Orcetra's performance advantage disappeared, indicating that the initial impressive results were inflated by methodological errors. AI
IMPACT Highlights critical issues in evaluating AutoML systems, potentially leading to more rigorous benchmarking standards.
RANK_REASON The cluster contains an academic paper detailing methodological flaws in AutoML system comparisons. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
