PulseAugur
EN
LIVE 06:57:54

Study reveals manuscript-level leakage inflates AI script classification accuracy

By PulseAugur Editorial · [1 sources] ·

A new paper explores evaluation protocols for historical script classification, highlighting the significant impact of manuscript-level leakage on accuracy metrics. Researchers found that testing on pages from training manuscripts drastically inflates perceived accuracy, with a drop from 97% to 54% when moving from page-level to manuscript-level splits. The study also investigated the role of soft labels and confidence calibration, suggesting that while soft labels may not improve accuracy, they can lead to predicted distributions closer to expert assessments and reduce divergence from expert evaluations. AI

IMPACT Highlights critical considerations for evaluating AI models in specialized domains, particularly concerning data leakage and confidence calibration.

RANK_REASON Academic paper detailing evaluation protocols and findings for a specific AI task. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →

Study reveals manuscript-level leakage inflates AI script classification accuracy

How we ranked this

Signal score
37 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Tool
Academic paper detailing evaluation protocols and findings for a specific AI task. [lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration
Single-source cluster
Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics
paper, other
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
Breaking (< 6h)
Fresh story with cross-source coverage still developing. Ranking may shift as more sources report.

Full methodology in our editorial standards.

COVERAGE [1]

  1. arXiv cs.CV TIER_1 English(EN) · Daniil I. Naryzhnyi (ITMO University), Roman A. Popkov (ITMO University), Daniil A. Bakalin (ITMO University) ·

    Evaluation protocols for historical script classification: the cost of manuscript-level leakage and the role of soft labels

    arXiv:2611.12969v1 Announce Type: new Abstract: A paleographer applies a script type classifier to unseen manuscripts. We measure how much testing on other pages of training manuscripts inflates accuracy. On VML-HP-ext (715 pages, 171 Hebrew manuscripts, 14 script classes) we com…

RELATED ENTITIES

RELATED TOPICS