Prediction-Powered Risk Monitoring of Deployed Models for Detecting Harmful Distribution Shifts
Researchers have developed a new semi-supervised method called Prediction-Powered Risk Monitoring (PPRM) to track model performance in environments with scarce labeled data. PPRM combines synthetic labels with a small set of true labels to create lower bounds on the running risk. This approach allows for the detection of harmful distribution shifts by comparing these bounds to an upper bound on nominal risk, offering finite-sample guarantees on type-I errors. The method has been validated through experiments in image classification, large language models, and telecommunications monitoring. AI
IMPACT Provides a novel approach for detecting performance degradation in AI models, crucial for maintaining safety and reliability in dynamic environments.