New metric quantifies LLM misguidance from buggy code in unit tests

A new paper introduces a metric to quantify the "misguidance effect," where buggy code prompts Large Language Models (LLMs) to generate unit tests that validate incorrect behavior instead of exposing it. This effect leads to more misguided tests and fewer effective ones. To address this, the researchers propose a specification-based prompting method that uses LLM-generated docstrings instead of the buggy code itself, significantly improving test generation for both buggy and bug-free code. AI

IMPACT This research could improve the reliability of LLM-generated code, making them more useful for software development.

RANK_REASON Academic paper introducing a new metric and methodology for evaluating LLM-generated unit tests. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• buggy code
• Large Language Models
• LLM-Generated Unit Tests

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →