MLCommons
PulseAugur coverage of MLCommons — every cluster mentioning MLCommons across labs, papers, and developer communities, ranked by signal.
-
MLCommons launches benchmark to test LLM safety against jailbreak attacks
MLCommons has released the Jailbreak Benchmark v1.0, a new methodology for assessing the robustness of large language models (LLMs) against adversarial prompts designed to bypass safety safeguards. The benchmark evaluat…
-
Granite Guardian Outperforms Llama Guard in Multilingual and Regulated LLM Safety
Llama Guard and Granite Guardian are two open-weight safety classifiers for LLMs, but they differ significantly in their capabilities and licensing. While Llama Guard excels at content safety and is suitable for English…
-
AI agents' security and privacy risks detailed in new research and workshop
Multiple research papers and a Google workshop report highlight the critical need for enhanced security and privacy measures in autonomous AI agents. These works address challenges such as unstructured input ambiguity, …
-
Google DeepMind pilots double-blind AI evaluations to prevent model cheating
Google DeepMind has developed a novel method for conducting double-blind evaluations of advanced AI models to prevent cheating by AI agents or developers. This approach utilizes Google Cloud's Confidential Computing to …
-
First double-blind evaluation of proprietary AI model conducted
A significant collaboration between AVERI, Google DeepMind, OpenMined, and MLCommons has resulted in the first-ever double-blind evaluation of a proprietary language model. This milestone involved both technical and ins…
-
AI benchmarking tools: Custom builds vs. existing solutions explored
The author of a blog focused on AI hardware and model performance investigated whether their custom-built benchmarking tools were necessary or if existing solutions could have been utilized. They found that while hardwa…
-
Amazon Nova 2 Lite uses new prompts for content moderation
Amazon has detailed how its Nova 2 Lite model can be effectively prompted for content moderation tasks. A new technical report and blog post explain how structured, taxonomy-driven prompting, aligned with the MLCommons …
-
OpenSSF launches AI/ML security working group
The Open Source Security Foundation (OpenSSF) has launched a working group focused on the intersection of AI/ML and security. This group aims to explore the security risks associated with AI technologies like LLMs and G…
-
OpenAI unveils GPT-5.6, expands ChatGPT Ads, and explores enterprise AI adoption · 10 sources tracked
OpenAI has announced GPT-5.6, a new frontier intelligence model emphasizing increased intelligence per token, improved performance per dollar, and enhanced on-demand capabilities for complex tasks. This release follows …
-
MLC enables running large models on browsers, iPhones, and AMD cards
The Machine Learning Compilation (MLC) group, led by Tianqi Chen at CMU, is developing frameworks like MLC Chat and Web LLM to enable running large language models on consumer hardware, including iPhones and web browser…