A new paper published on arXiv introduces a critical finding regarding the evaluation of Out-of-Distribution (OOD) detection in Evidential Deep Learning (EDL). The research demonstrates that the common metric of 'vacuity' is highly sensitive to differences in class cardinality between in-distribution and OOD datasets. This sensitivity can artificially inflate evaluation scores like AUROC and AUPR, even when model predictions remain unchanged. The paper argues for more precise definitions of ID and OOD, particularly when evaluating EDL on causal language models with MCQA datasets. AI
影响 Highlights a significant evaluation artifact in OOD detection for EDL models, potentially impacting benchmark reliability and model comparisons.
排序理由 The cluster contains a new academic paper detailing a novel finding in AI evaluation methodology.
- arXiv
- causal language models
- Multiple-Choice Question-Answer datasets
- AUROC
- Out-of-Distribution detection
- Evidential Deep Learning
AI 生成摘要 · Google Gemini · 来自 2 个来源。 我们如何撰写摘要 →
