New benchmark tests LLMs for quantum code version compatibility

A new benchmark, quantum-api-drift, has been developed to evaluate how well large language models can generate quantum code that is compatible with specific software development kit (SDK) versions. The benchmark was tested using Qiskit across versions v0.43, v1.3, and v2.0, with 17 models evaluated on 50 tasks. Claude Opus 4.7 performed best on v0.43 and v2.0, while Grok 4.20 excelled on v1.3. The study found that while documentation-guided repair can help, API drift remains a significant challenge in LLM-generated quantum code. AI

IMPACT Highlights the need for LLMs to maintain version fidelity in code generation, impacting future development tools.

RANK_REASON Academic paper introducing a new benchmark for LLM-generated code. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• Claude Opus 4.7
• Codex CLI
• GPT-5.4
• Grok 4.20
• Qiskit
• v0.43
• v2.0

• paper
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →