Beyond Accuracy: Measuring Logical Compliance of Predictive Models
Researchers have introduced a new metric called the Rule Violation Score (RVS) to evaluate the logical compliance of predictive models, which goes beyond traditional accuracy measures. RVS quantifies how well a model adheres to predefined logical or domain-specific constraints, treating hard and soft rules distinctly. This metric can be applied to any dataset and model type, and it can even assess the logical consistency of training datasets and identify poorly defined rules. Experiments on knowledge graph link prediction and relational regression benchmarks showed that models with similar predictive accuracy can have significantly different logical compliance, a distinction missed by standard metrics. AI
IMPACT Introduces a novel metric to assess model adherence to logical rules, crucial for high-stakes AI applications beyond simple accuracy.