A new audit of the RVL-CDIP benchmark for document classification has revealed significant data quality issues, including approximately 10.5% of documents with incorrect labels and 35.2% of the test set having duplicates in the training set. These flaws impact model performance, with state-of-the-art models losing accuracy when identification codes are removed or when trained on documents from other sources. The audit also identified nearly 16,000 documents containing sensitive personal data, for which de-identified versions have been produced. After remediating these defects and retraining models, the most accurate model, LayoutLMv3, achieved 96.5% accuracy, an improvement of 0.9 to 2.9 points over its original condition. AI
IMPACT Highlights the critical need for robust data quality in AI benchmarks, impacting model reliability and generalizability.
RANK_REASON The item is an academic paper detailing a systematic audit of a benchmark dataset. [lever_c_demoted from research: ic=1 ai=1.0]
- alphaXiv
- arXiv
- CatalyzeX
- Connected Papers
- DagsHub
- Gotit.pub
- Hugging Face
- LayoutLMv3
- Litmaps
- RVL-CDIP
- ScienceCast
- scite
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
