Researchers have developed ProofJudge, a new system that uses a large language model (LLM) to evaluate the quality of formal proofs in the Lean 4 programming language, specifically within the Mathlib library. This agentic system goes beyond mere correctness to assess proofs on dimensions such as library leverage, automation fit, structural clarity, statement quality, and adherence to Mathlib conventions. ProofJudge was evaluated on a dataset of 218 pull requests and demonstrated an ability to align with human reviewer preferences, with some open-weight models achieving this alignment at a significantly lower cost than proprietary options. The project aims to foster further research by releasing the judge harness, evaluation dataset, and traces as open-source artifacts. AI
IMPACT Enhances the rigor and efficiency of formal verification in mathematics and computer science by automating proof quality assessment.
RANK_REASON The item describes a new research paper detailing a novel evaluation system for formal proofs. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
