Formalize, Don't Optimize: The Heuristic Trap in LLM-Generated Combinatorial Solvers
Researchers have developed a new benchmark, CP-SynC-XL, comprising 100 combinatorial problems to evaluate how Large Language Models (LLMs) synthesize executable solvers. Their findings indicate that using LLMs to formalize problems for existing solvers like OR-Tools in Python yields higher correctness than declarative modeling in MiniZinc. Prompting LLMs to also optimize search strategies resulted in only minor speed-ups and a significant drop in correctness for many problems, attributed to a "heuristic trap" where LLMs replace complete search with approximations or introduce over-constraining machinery. AI
IMPACT Highlights the risks of using LLMs for direct optimization in solver generation, suggesting a focus on formalization for verified solvers.