New RepoExec benchmark reveals LLMs struggle with code dependency accuracy

A new benchmark called RepoExec, introduced in a paper from FPT Software AI Center, evaluates Large Language Models on code generation by considering both correctness and dependency invocation rate. The benchmark revealed that current models struggle to accurately utilize provided dependencies, with many models failing to call the correct dependencies or rewriting existing functions instead of solving the problem. The research also found that providing only function signatures and docstrings, without the function body, can lead models to misinterpret the task as a few-shot example, resulting in empty or incorrect code. AI

IMPACT Highlights a critical gap in LLM code generation capabilities, suggesting current models may not reliably integrate external code dependencies.

RANK_REASON Research paper introducing a new benchmark for LLM code generation. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

• Bleu
• Dependency Invocation Rate
• Dung Manh Nguyen
• FPT Software AI Center
• GPT-4o mini
• NAACL 2025 Findings
• Nam Le Hai
• Nghi D. Q. Bui
• Python
• RepoExec

• paper
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →