研究人员开发了一种方法，可以从原始维基媒体转储中为七种南斯拉夫语创建高质量的训练语料库。该过程包括两个主要阶段：从各种维基百科项目中提取和清理文本，然后使用基于n-gram的策略过滤掉低质量或重复的文章。这种方法旨在生成适合训练语言模型和进行比较语言学研究的语言丰富的数据集，并有可能推广到其他语言。 AI
影响 提供了一种生成专业语言语料库的可扩展方法，有可能提高大型语言模型在资源匮乏语言上的性能。
排序理由 详细介绍创建训练数据方法的学术论文。
AI 生成摘要 · Google Gemini · 来自 3 个来源。 我们如何撰写摘要 →
arXiv:2604.25384v1 Announce Type: new Abstract: This paper presents a methodology for transforming raw Wikimedia dumps into quality textual corpora for seven South Slavic languages. The work is divided into two major phases. The first involves extracting and cleaning text from ra…
This paper presents a methodology for transforming raw Wikimedia dumps into quality textual corpora for seven South Slavic languages. The work is divided into two major phases. The first involves extracting and cleaning text from raw dumps of Wikipedia, Wikisource, Wikibooks, Wik…
This paper presents a methodology for transforming raw Wikimedia dumps into quality textual corpora for seven South Slavic languages. The work is divided into two major phases. The first involves extracting and cleaning text from raw dumps of Wikipedia, Wikisource, Wikibooks, Wik…