New APR evaluation index reveals trade-offs in dense vs MoE models

A new research paper introduces a multi-dimensional evaluation framework for automated program repair (APR) models, moving beyond simple test-passing metrics. The proposed Weighted Quality Index (QI), inspired by ISO/IEC 25010, incorporates functional correctness, maintainability, security, and generation efficiency. When applied to Qwen2.5-Coder and DeepSeek-Coder-V2 Lite models on bug datasets, the study found that model rankings shifted based on the QI's weighting schemes, highlighting trade-offs often missed by single-metric evaluations. Notably, the DeepSeek-Coder-V2 Lite Mixture-of-Experts (MoE) model demonstrated comparable correctness to larger dense models while using significantly fewer active parameters, suggesting active parameter count is a more relevant metric for sparse code models. AI

IMPACT Introduces a more nuanced evaluation for code generation models, potentially guiding future development and benchmarking.

RANK_REASON Research paper proposing a new evaluation methodology for AI models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• DeepSeek Coder V2 Lite
• Defects4J
• Hugging Face
• ISO/IEC 25010
• QuixBugs
• Qwen2.5 Coder

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →