New method adapts English LLM data classifiers for multilingual pretraining

Researchers have developed a method to adapt English-based quality classifiers for selecting high-quality pretraining data for multilingual large language models (LLMs). This approach involves training a small multi-layer perceptron on top of Transformer encoder embeddings, using machine-translated text and scores from English classifiers as labels. Experiments across various model scales demonstrate that this multilingual adaptation maintains downstream LLM benchmark performance without compromising regional and cultural knowledge. AI

IMPACT Enables more effective and efficient pretraining of multilingual LLMs by leveraging existing English data quality classifiers.

RANK_REASON Academic paper detailing a new methodology for LLM pretraining data selection. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• English
• Hugging Face
• large language model
• multilayer perceptron
• Transformer

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →