A new research paper explores the effectiveness of fine-tuning large language models for the neural decompilation of Dart Ahead-of-Time (AOT) binaries. The study found that fine-tuning did not significantly improve pass@k performance and, in some cases, led to regressions. Researchers also observed metric divergence, where metrics like CodeBLEU and compile@k improved while pass@k declined, suggesting that fine-tuning may optimize for superficial similarity rather than functional correctness. The paper introduces the HumanEval-Dart benchmark and advocates for pass@k as the primary evaluation metric for neural decompilation tasks. AI
IMPACT Suggests that current fine-tuning methods may not improve functional correctness in code generation tasks, highlighting the need for better evaluation metrics.
RANK_REASON Research paper published on arXiv detailing empirical study of LLM fine-tuning and evaluation metrics for code generation.
AI-generated summary · Google Gemini · from 2 sources. How we write summaries →
