A new study published on arXiv evaluates the performance and memorization tendencies of large language models (LLMs) when writing TLA+ specifications. The research introduces a method using consistent identifier renaming (CIR) to test for memorization, finding that renaming identifiers significantly decreases the correctness scores of models like Claude Opus 4.8 and GPT-5 on TLA+-Bench tasks. However, when grading is based on preserved behavior rather than exact names, the performance drop is substantially reduced, suggesting that current benchmarks may conflate memorization with name-dependent grading and the cost of learning new names. AI
IMPACT Highlights potential flaws in LLM evaluation benchmarks, suggesting a need for more robust methods that distinguish true understanding from memorization.
RANK_REASON Academic paper detailing a new evaluation method for LLMs. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
