Study reveals LLM judges exhibit biases in code evaluation

A new study published on arXiv explores the biases present in large language models (LLMs) when they are used to evaluate code. The research identifies six types of potential bias, demonstrating that LLM judges can unfairly inflate or deflate scores for semantically equivalent code due to superficial variations like variable names or formatting. Even when prompted to generate test cases beforehand, the LLM judges remained susceptible to these biases across multiple programming languages and models, indicating a need for more robust code evaluation methods. AI

IMPACT Highlights potential flaws in automated code assessment, suggesting current LLM judges may not be reliable for evaluating code quality.

RANK_REASON The cluster contains a research paper published on arXiv detailing findings about LLM biases. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• code evaluation
• Hugging Face
• Jiwon Moon
• large language models
• LLM judges

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →