Brief

A new paper published on arXiv demonstrates that cross-validation, a common statistical technique for comparing machine learning models, can produce unstable and invalid inferences. The research specifically highlights that the Lasso and soft-thresholding methods, despite being individually stable, can lead to unreliable comparisons. This instability calls into question the routine use of cross-validation for model comparison without prior verification of relative stability. AI

Researchers have developed a new framework called Target-Weighted Cross-Validation (TWCV) to improve the accuracy of performance estimates for spatial prediction models. This method addresses the common issue where validation data does not accurately reflect the conditions under which the model will be deployed, leading to biased results. TWCV uses spatially relevant data like environmental covariates and prediction distances to align validation tasks with real-world deployment scenarios. Simulations and a case study on mapping nitrogen dioxide concentrations in Germany showed that TWCV significantly reduces bias compared to standard cross-validation techniques, providing more reliable error estimates. AI