New benchmark evaluates LLMs on TLA+ spec generation by execution

Researchers have introduced TLA$^{+}$-Bench, a new benchmark and dataset designed to more accurately evaluate the performance of large language models in generating TLA$^{+}$ formal specifications from natural language. Unlike previous methods that relied on resemblance or parseability, TLA$^{+}$-Bench grades specifications based on execution using a TLA$^{+}$ model checker. The dataset includes 403 model-checked specifications and 897 parse-only specifications, along with various labels for difficulty and category. A key finding is that the correctness rate of model-generated TLA$^{+}$ specifications can vary significantly based on grading choices, with rates ranging from 1.7% to 26% depending on the evaluation method and whether interface names are provided to the model. AI

IMPACT This benchmark could lead to more reliable evaluation of LLMs for formal specification generation, improving software engineering practices.

RANK_REASON The item is a research paper introducing a new benchmark and dataset for evaluating LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• large-language models
• model checker
• natural language
• Open Models
• TLA$^{+}$-Bench

• paper
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →