Lara Mini Base model boosts multilingual performance with MoE architecture

Researchers have introduced Lara Mini Base, a new language model designed to improve performance across multiple languages without sacrificing capacity. Unlike most models trained predominantly on English, Lara Mini Base was trained on 6 trillion tokens encompassing English and 44 other languages, including all official EU languages. The model utilizes a Mixture of Experts (MoE) architecture with 7.2 billion total parameters and 0.6 billion active parameters, which helps reduce capacity competition between languages. This approach allows Lara Mini Base to achieve high scores on English, multilingual, math, and code benchmarks, outperforming models trained on significantly more data, particularly in multilingual and math tasks. AI

IMPACT This model's approach to multilingual training could lead to more equitable and performant AI systems across diverse language communities.

RANK_REASON The cluster describes a new language model presented in an arXiv paper, detailing its architecture and performance on benchmarks. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International
• English
• European Union
• HellaSwag
• Hugging Face
• Lara Mini base
• T-HellaSwag

• model release
• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →