LLM autoformalization struggles with paraphrased inputs

Researchers have investigated the robustness of large language models (LLMs) in autoformalization tasks, specifically their ability to generate formal proofs from natural language statements. The study found that LLMs exhibit performance variability when presented with semantically similar paraphrased inputs, indicating that minor alterations in the natural language can significantly affect the generated formal outputs. This research utilized the MiniF2F and Lean 4 ProofNet benchmarks to evaluate two modern LLMs, measuring both semantic and compilation validity of the generated proofs. AI

IMPACT Highlights LLM sensitivity to input phrasing, suggesting a need for more robust natural language understanding in formal reasoning tasks.

RANK_REASON This is a research paper published on arXiv detailing experimental findings on LLM robustness. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• Hayden Moore
• Large Language Models
• Lean 4 ProofNet
• MiniF2F

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →