LLMs hallucinate non-existent libraries in code generation, study finds

A new study published on arXiv details the risks of large language models (LLMs) hallucinating non-existent libraries when generating code. Researchers found that variations in developer prompts, including misspellings and fabricated library names, can trigger these hallucinations at high rates, potentially leading to broken builds and security vulnerabilities. To address this, the study introduces LibHalluBench, a benchmark for systematically evaluating these library hallucinations and highlights the urgent need for safeguards. AI

IMPACT Highlights systemic vulnerabilities in LLMs for code generation, necessitating safeguards against library hallucinations and supply chain risks.

RANK_REASON The cluster contains a research paper detailing a systematic study and introducing a new benchmark for evaluating a specific failure mode in LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• alphaXiv
• arXiv
• CatalyzeX Code Finder for Papers
• CORE Recommender
• DagsHub
• Gotit.pub
• Hugging Face
• LibHalluBench
• Library Hallucinations
• LLMs
• Lukas Twist
• ScienceCast

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →