New multilingual model NE-BERT boosts NLP for 9 Northeast Indian languages

Researchers have developed NE-BERT, a new multilingual language model specifically designed for nine underrepresented Northeast Indian languages. This model, trained on approximately 8.3 million sentences, significantly outperforms existing models like IndicBERT-V2 and MuRIL in perplexity and tokenization for these low-resource languages. NE-BERT addresses vocabulary fragmentation issues through aggressive upsampling and has demonstrated practical utility in downstream tasks such as part-of-speech tagging, with its code and data released to foster further NLP research and digital inclusion. AI

IMPACT Enhances NLP capabilities for underrepresented languages, potentially enabling new applications and digital inclusion.

RANK_REASON The cluster describes a new academic paper detailing the creation and evaluation of a specialized language model. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• English
• Hindi
• Hugging Face
• IndicBERT-V2
• Kokborok
• multilingual-BERT
• NE-BERT
• Northeast Indian languages
• Pnar

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →