New research reveals flaws in generative time-series model evaluation

A new research paper published on arXiv highlights critical flaws in the evaluation protocols for generative time-series models. The study reveals that standard benchmarking methods can misrepresent model performance by using evaluation windows with vastly different data distributions than the training sets. Furthermore, the paper introduces a control method to isolate the impact of temporal coupling on statistics and demonstrates that an autoregressive hurdle model outperforms a conditional flow model on most datasets, despite the latter's variability across training seeds. The research also points out that model rankings change depending on the occurrence statistics used, indicating a need for more robust and consistent evaluation practices. AI

IMPACT Highlights critical issues in evaluating generative models, potentially leading to more reliable benchmarks and improved model development.

RANK_REASON Academic paper detailing new findings and methodologies in AI model evaluation. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• alphaXiv
• arXiv
• CatalyzeX
• DagsHub
• Generative Time-Series Models
• Gotit.pub
• Hugging Face
• IArxiv
• Influence Flower
• ScienceCast

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →