PulseAugur
EN
LIVE 09:05:56

LLM-generated code proofs show brittleness under refactoring · arXiv research

By PulseAugur Editorial · [1 sources] ·

A new study titled "How Stable Are LLM-Generated Proofs Under Program Refactoring?" introduces RefactorProof, a benchmark designed to test the resilience of code generation proofs after program refactoring. The benchmark comprises 458 Lean-certified variants derived from 185 VERINA tasks, maintaining the original specification and proof. Results indicate significant brittleness in proofs generated by both reference systems and specialized LLM provers, with failure rates ranging from 4.3% to 26.8%. Despite this, the specialized provers demonstrated higher stability than reference proofs, with DeepSeek-Prover-V2-7B, Goedel-Prover-V2-32B, and Goedel-Prover-V2-8B showing improved unchanged-proof survival rates. AI

IMPACT Highlights the need for more robust LLM-generated code proofs that can withstand program refactoring, impacting the reliability of AI-assisted software development.

RANK_REASON The cluster contains an academic paper detailing a new benchmark and experimental results for LLM-generated code proofs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →

LLM-generated code proofs show brittleness under refactoring · arXiv research

How we ranked this

Signal score
16 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Tool
The cluster contains an academic paper detailing a new benchmark and experimental results for LLM-generated code proofs. [lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration
Single-source cluster
Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics
paper, other
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
Breaking (< 6h)
Fresh story with cross-source coverage still developing. Ranking may shift as more sources report.

Full methodology in our editorial standards.

COVERAGE [1]

  1. arXiv cs.AI TIER_1 English(EN) · Aravind Mandiga, Guoming Li ·

    How Stable Are LLM-Generated Proofs Under Program Refactoring? A Stress Test for Verifiable Code Generation

    arXiv:2610.36328v1 Announce Type: new Abstract: Benchmarks for verifiable code generation usually test a proof against one fixed implementation. We ask whether the same proof remains valid after a program refactoring that preserves behavior. We introduce RefactorProof, which deri…

RELATED ENTITIES

RELATED TOPICS