OpenLID-v3: Improving the Precision of Closely Related Language Identification -- An Experience Report
Researchers have developed OpenLID-v3, an enhanced language identification system designed to improve the accuracy of distinguishing closely related languages and filtering out noise from web data. The updated system incorporates more training data, merges problematic language variant clusters, and introduces a specific label for noise detection. Evaluations against existing tools like GlotLID on various benchmarks, with a focus on language groups such as Slavic, Romance, and Scandinavian languages, indicate that while ensemble approaches boost precision, they can reduce coverage for low-resource languages. The OpenLID-v3 system and its associated datasets are now publicly available. AI