LLMs struggle with South Asian music generation and understanding

Researchers have evaluated the capabilities of Large Language Models (LLMs) in understanding and generating South Asian classical music, a domain with distinct structural principles from Western traditions. Their new benchmark, comprising 504 questions, tested 33 LLMs, with top models like Gemini 2.5 Pro achieving high accuracy in understanding, while most open-source models performed poorly. For music generation, even the best models only produced stylistically faithful outputs 40% of the time, indicating that structural validity and stylistic faithfulness are separate challenges for AI in this low-resource musical context. AI

IMPACT Highlights limitations of current LLMs in culturally specific, low-resource domains, indicating a need for more specialized models.

RANK_REASON Academic paper introducing a new benchmark and evaluation of LLMs on a specific domain. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• Gemini 2.5 Pro
• Large Language Models

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →