研究人员审计了伦巴第语的文本语料库，揭示了数据质量和代表性方面存在严重问题。尽管网络抓取的数据看起来很丰富，但许多数据集存在识别错误、样板文本和非语言噪音。分析还突显了对西部伦巴第语方言的严重偏见，边缘化了东部方言，并表明需要社区驱动、顾及方言差异的数据整理，而不是简单的基于数量的抓取。 AI
影响 凸显了资源匮乏语言在数据质量和代表性方面面临的关键挑战，影响了自然语言处理模型的发展。
排序理由 该集群包含一篇详细介绍语言语料库研究结果的学术论文。
AI 生成摘要 · Google Gemini · 来自 2 个来源。 我们如何撰写摘要 →
arXiv:2606.06349v1 Announce Type: new Abstract: Several of the world's languages are still under-resourced in terms of Natural Language Processing (NLP) tools. This is mostly due to the lack of high-quality datasets to train, develop, and evaluate systems and models for several t…
Several of the world's languages are still under-resourced in terms of Natural Language Processing (NLP) tools. This is mostly due to the lack of high-quality datasets to train, develop, and evaluate systems and models for several tasks, such as Machine Translation (MT). We condu…