Researchers have developed a new metric called the Harmful Memorization Index (HMI) to evaluate how Large Language Models (LLMs) handle code generation, specifically focusing on problematic memorization rather than benign code reuse. The HMI assesses a model's tendency to produce code highly similar to original solutions even when the task semantics are altered, leading to performance degradation. Experiments on MBPP+ and BigCodeBench datasets revealed that model scale does not consistently correlate with HMI, and supervised fine-tuning can increase HMI while Proximal Policy Optimization offers a better accuracy-HMI trade-off. AI
IMPACT Introduces a new metric to better understand LLM brittleness and potential security risks in code generation.
RANK_REASON The cluster is about a research paper introducing a new metric and methodology for evaluating LLMs. [lever_c_demoted from research: ic=1 ai=1.0]
- arXiv
- BigCodeBench
- Harmful Memorization Index
- Hugging Face
- LLM Code Generation
- MBPP+
- Proximal Policy Optimization
- supervised fine-tuning
- Wentao Chen
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
