MLCommons has released the Jailbreak Benchmark v1.0, a new methodology for assessing the robustness of large language models (LLMs) against adversarial prompts designed to bypass safety safeguards. The benchmark evaluates eight open-weight systems using 264 seed prompts across eleven hazard categories. Results showed an increase in unsafe response rates from 11.08% under baseline conditions to 18.65% under jailbreak conditions, indicating an average "Resilience Gap" of 7.57%. The benchmark aims to provide a reproducible foundation for comparative jailbreak evaluations and future research. AI
IMPACT Establishes a standardized method for evaluating LLM safety against adversarial attacks, potentially driving improvements in model robustness.
RANK_REASON Publication of a new benchmark and methodology for evaluating LLM safety. [lever_c_demoted from research: ic=1 ai=1.0]
- AILuminate Assessment Standard v1.4
- arXiv
- Hugging Face
- large-language models
- MLCommons
- MLCommons Jailbreak Benchmark v1.0
- MLCommons Jailbreak Taxonomy
- Resilience Gap
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
