PulseAugur
EN
LIVE 08:17:32

LLMs show limited performance gains in screening software engineering papers

By PulseAugur Editorial · [1 sources] ·

A new study published on arXiv evaluates the performance of eight large language models (LLMs) in screening scientific papers for software engineering systematic reviews. The research found that while newer LLMs showed a marginal improvement in performance compared to older models, the differences were not substantial. LLMs still struggle to replace human reviewers for this task, with agreement between models being high but significant disagreements arising on specific inclusion and exclusion criteria. Refining these criteria offered only modest improvements, suggesting future research should focus on agent-based approaches and prompt engineering. AI

IMPACT LLMs show limited progress in automating systematic review screening, indicating human oversight remains crucial.

RANK_REASON Research paper evaluating LLM performance on a specific task. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →

LLMs show limited performance gains in screening software engineering papers

How we ranked this

Signal score
18 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Tool
Research paper evaluating LLM performance on a specific task. [lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration
Single-source cluster
Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics
paper, other
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
Breaking (< 6h)
Fresh story with cross-source coverage still developing. Ranking may shift as more sources report.

Full methodology in our editorial standards.

COVERAGE [1]

  1. arXiv cs.AI TIER_1 English(EN) · Aleksi Huotala, Miikka Kuutila, Mika M\"antyl\"a ·

    Has LLM Screening Performance Stalled in Software Engineering Systematic Reviews?

    arXiv:2610.10633v1 Announce Type: cross Abstract: Screening in systematic reviews (SRs) is manual and time-consuming. Prior work has explored large language models (LLMs) for automating this step, but LLMs are evolving rapidly, so earlier performance claims may no longer accurate…

RELATED ENTITIES

RELATED TOPICS