An experiment using Qwen2.5-Coder models locally revealed that repeated code reviews by LLMs, especially with sampling enabled, tend to converge on persistent findings rather than accurate ones. When run at temperature 0, the models produced identical results, but increasing the temperature to 0.7 introduced significant variability. Majority voting on these varied findings often selected consistent but incorrect bug reports over less frequent but accurate ones, highlighting a potential pitfall in using LLM-generated code reviews. AI
IMPACT Highlights a potential flaw in using LLM-generated code reviews, suggesting that majority voting may select persistent errors over correct findings.
RANK_REASON The item details an experiment and its findings regarding the behavior of LLMs in code review tasks. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
