GitBug-Java: A Reproducible Benchmark of Recent Java Bugs

LLMs show significant performance drops on transformed benchmarks, indicating memorization

Researchers have developed a new method combining metamorphic testing with negative log-likelihood to diagnose data leakage in large language models used for program repair. By creating variant benchmarks through semant…