Can we trust our models? Epistemic calibration in second-order classification
Researchers have introduced a new metric called epistemic calibration to assess the trustworthiness of uncertainty estimates in machine learning models. This metric goes beyond classical calibration by evaluating whether the reported epistemic uncertainty accurately reflects the spread of model predictions relative to the true values. The proposed Expected Epistemic Calibration Error (EECE) serves as a consistent estimator for this new criterion, and experiments demonstrate its effectiveness in distinguishing between various uncertainty quantification methods. AI
IMPACT Introduces a novel method for evaluating AI model reliability, crucial for high-stakes applications.