New research questions random splits for CTR model calibration

A new paper evaluates the effectiveness of post-hoc calibration for click-through rate (CTR) models, challenging the common practice of using random data splits for offline evaluation. The research, conducted on the Avazu dataset, found that temporal splits and recalibration methods like Platt scaling and isotonic regression can significantly reduce calibration errors. However, the optimal fitting window for recalibration is crucial, as too short a window can increase errors, while too long a window may not yield optimal results. The study suggests that a 12-hour fitting slice, even when subsampled, offers a good balance of performance and efficiency, and recommends temporal evaluation protocols for more accurate offline assessment. AI

IMPACT Challenges standard offline evaluation practices for CTR models, potentially leading to more accurate real-world performance predictions.

RANK_REASON Paper published on arXiv detailing new evaluation methods for ML models.

Read on arXiv cs.LG →

• arXiv
• Avazu Inc
• DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems
• Hugging Face
• Isotonic regression
• logistic regression model
• Platt scaling
• alphaXiv
• CatalyzeX
• DagsHub
• Expected Calibration Error (ECE)
• Gotit.pub
• IArxiv
• ScienceCast

• paper
• model release

AI-generated summary · Google Gemini · from 2 sources. How we write summaries →