Open-weight LLMs show promise but fall short in PDDL model repair

Researchers have evaluated the effectiveness of open-weight large language models in repairing errors within PDDL (Planning Domain Definition Language) models, which are crucial for AI planning. Their experiments demonstrated that while the best-performing LLM achieved an F1 score of 0.87, significantly outperforming a symbolic baseline, this performance came with a low test pass rate. Even with test traces included, the models struggled to consistently satisfy the constraints required for reliable automated model repair, particularly on complex domains. AI

IMPACT Current open-weight LLMs are not yet reliable enough for automated AI planning model repair, indicating a need for further development in reasoning and constraint satisfaction.

RANK_REASON Research paper evaluating LLM capabilities on a specific AI task. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• LLM
• open-weight models
• Thoughtful domain

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →