PulseAugur
EN
LIVE 09:14:19

LLM safety refusals found unstable across random seeds and temperatures

By PulseAugur Editorial · [1 sources] ·

A new research paper highlights significant inconsistencies in the safety refusal behavior of large language models. The study found that between 18% and 28% of harmful prompts resulted in differing refusal decisions when tested with various random seeds and temperature settings. Increasing the temperature parameter was shown to decrease decision stability, with mean stability dropping from 0.977 at temperature 0.0 to 0.942 at temperature 1.0. The findings suggest that current single-shot safety evaluation methods are insufficient and that evaluation protocols must account for the stochastic nature of LLM responses. AI

IMPACT Highlights the need for more robust LLM safety evaluation methodologies that account for stochastic variations in model output.

RANK_REASON Research paper detailing findings on LLM behavior. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →

LLM safety refusals found unstable across random seeds and temperatures

How we ranked this

Signal score
14 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Tool
Research paper detailing findings on LLM behavior. [lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration
Single-source cluster
Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics
paper, safety
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
Breaking (< 6h)
Fresh story with cross-source coverage still developing. Ranking may shift as more sources report.

Full methodology in our editorial standards.

COVERAGE [1]

  1. arXiv cs.CL TIER_1 English(EN) · Erik Larsen ·

    The Instability of Safety: How Random Seeds and Temperature Expose Inconsistent LLM Refusal Behavior

    arXiv:2512.12066v3 Announce Type: replace-cross Abstract: Current safety evaluations of large language models rely on single-shot testing, implicitly assuming that model responses are deterministic and representative of the model's safety alignment. We challenge this assumption b…

RELATED ENTITIES

RELATED TOPICS