Audit reveals major data quality flaws in standard RVL-CDIP benchmark

A new audit of the RVL-CDIP dataset, a standard benchmark for document classification, has revealed significant data quality issues. Researchers found that approximately 10.5% of documents have incorrect labels, and a substantial 35.2% of the test set contains duplicates or near-duplicates present in the training set. These flaws can lead to inflated accuracy scores and models that perform poorly on out-of-distribution data. The study also identified sensitive personal data in nearly 16,000 documents, for which de-identified versions have been produced. After remediating these defects, re-benchmarked models showed improved accuracy, with LayoutLMv3 achieving 96.5%. AI

IMPACT Identified data quality issues in RVL-CDIP could lead to more robust and reliable document classification models.

RANK_REASON The cluster contains a research paper auditing a widely used dataset, identifying significant flaws and releasing corrected versions.

Read on arXiv cs.CL →

• alphaXiv
• arXiv
• CatalyzeX
• Connected Papers
• DagsHub
• Gotit.pub
• Hugging Face
• LayoutLMv3
• Litmaps
• RVL-CDIP
• ScienceCast
• scite

• paper
• other

AI-generated summary · Google Gemini · from 2 sources. How we write summaries →