PulseAugur
EN
LIVE 12:42:47

LLMs evaluated on Scrum certification questions; Gemini 3 Flash leads accuracy

By PulseAugur Editorial · [2 sources] ·

A recent study evaluated the performance of three large language models—GPT-5 mini, Gemini 3 Flash, and DeepSeek Chat 3.2—on 993 Scrum certification-style questions. Gemini 3 Flash demonstrated the highest accuracy, while all models showed low intra-model variability. Performance varied by question format and topic, with models excelling in normatively explicit areas and single-answer multiple-choice questions, but struggling with multi-select and True/False formats, as well as more interpretive Scrum topics. The analysis revealed systematic errors, including overgeneralization and conflicts between common interpretations and strict Scrum definitions. AI

IMPACT LLM performance on domain-specific certification questions varies, highlighting the need for careful prompting and evaluation for reliable use in professional training.

RANK_REASON The cluster consists of two academic papers presenting empirical research on LLM performance on a specific domain.

Read on arXiv cs.AI →

AI-generated summary · Google Gemini · from 2 sources. How we write summaries →

LLMs evaluated on Scrum certification questions; Gemini 3 Flash leads accuracy

How we ranked this

Signal score
0 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Research
The cluster consists of two academic papers presenting empirical research on LLM performance on a specific domain.
Source corroboration
2 independent sources
Multiple independent publishers reporting the same story raises confidence that it's real and newsworthy.
Topics
paper, product
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
100 days old
Aged out of breaking-news scoring windows; ranking reflects the durable signal from the full source set.

Full methodology in our editorial standards.

COVERAGE [2]

  1. arXiv cs.AI TIER_1 English(EN) · Robson Alves Vilar, Emanuel Dantas Filho, Ademar Fran\c{c}a de Sousa Neto, Mirko Perkusich, Danyllo Wagner Albuquerque, Jo\~ao Paiva, Kyller Gorg\^onio, Angelo Perkusich ·

    Comparing Large Language Models on Scrum Certification-Style Questions: Accuracy, Stability, and Error Patterns

    arXiv:2607.00048v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly used in exam- and certification-style question answering tasks, where their ability to retrieve, interpret, and apply domain-specific knowledge can be systematically assessed. In Softw…

  2. arXiv cs.AI TIER_1 English(EN) · Mirko Perkusich, Danyllo Albuquerque, Jo\~ao Paiva, Robson Vilar, Emanuel Dantas, Ademar Fran\c{c}a de Sousa Neto, Rohit Gheyi, Kyller Gorg\^onio, Angelo Perkusich ·

    Prompting GPT-5 on Scrum Certification Questions: An Empirical Accuracy Study

    arXiv:2607.00049v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly used in Agile Software Development for documentation, coaching, and training. As practitioners adopt these tools to prepare for certifications such as Professional Scrum Master (PSM), …

RELATED ENTITIES

RELATED TOPICS