LLMs struggle with semantic recall in long code contexts, new paper finds

A new paper published on arXiv explores the limitations of large language models (LLMs) in understanding long code contexts. Researchers found that while LLMs excel at lexical recall (verbatim code retrieval), their semantic recall (understanding operational semantics) significantly degrades when code is positioned in the middle of long inputs. The study introduces a metric called semantic recall sensitivity and proposes a new task, SemTrace, to better evaluate this capability. Findings suggest current benchmarks may overestimate LLMs' code understanding abilities. AI

IMPACT Highlights potential overestimation of LLM code understanding capabilities, suggesting a need for more robust evaluation methods.

RANK_REASON Academic paper detailing a new evaluation method and findings for LLM code understanding. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• Adam Štorek
• alphaXiv
• arXiv
• CatalyzeX
• CruxEval
• DagsHub
• Gotit.pub
• Hugging Face
• ScienceCast
• SemTrace

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →