Indian AI models strong on old benchmarks, lag in new evaluations

A new paper assesses the progress of Indian foundation models by analyzing publicly reported benchmark results. While Indian models show strong performance on established benchmarks like MMLU and MATH-500, they lag in participation in newer, more specialized evaluations. The study proposes a Benchmark Maturity Index (BMI) to evaluate the standardization, participation, verification, and national coverage of benchmarks, suggesting that apparent capability gaps may stem from evaluation ecosystem deficiencies. Sarvam AI is noted for having the broadest benchmark coverage among the surveyed Indian organizations. AI

IMPACT Highlights potential gaps in evaluating national AI capabilities and suggests criteria for funding and monitoring AI programs.

RANK_REASON Academic paper analyzing AI model benchmarks. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• Massive Multitask Language Understanding
• Math-500
• Sarvam AI

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →