Exploiting Similarities in A/B Testing with Off-Policy Estimation
Researchers have developed a new family of estimators for A/B testing that can improve statistical efficiency by exploiting similarities between the systems being compared. Traditional A/B testing treats systems as black boxes, but this new approach leverages off-policy estimation to account for shared structures and decision-making propensities. The proposed estimators are robust to misspecification and offer substantial accuracy gains when systems are similar, while gracefully defaulting to standard methods when they are not. AI
IMPACT Introduces a more statistically efficient method for evaluating system changes, potentially impacting how AI model performance is benchmarked.