Promptfoo
PulseAugur coverage of Promptfoo — every cluster mentioning Promptfoo across labs, papers, and developer communities, ranked by signal.
- 2026-05-20 product_launch Promptfoo integrates its attack plugins with the OWASP LLM Top 10 2025 security categories. source
6 day(s) with sentiment data
-
New tool muteval finds missing LLM tests by degrading systems
A new tool called muteval aims to address a critical blind spot in LLM testing by identifying tests that are missing rather than just reporting on existing ones that fail. Unlike traditional test runners, muteval intent…
-
TypeScript experiment compares agent trajectory checking tools
This item is a brief technical note discussing a small experiment in TypeScript. It compares two tools, AgentInspect and Promptfoo, for performing local checks on agent trajectories. The experiment focuses on the coding…
-
Promptfoo upgrades to GPT-5.6+, TRL trains beyond 1M tokens
Promptfoo has updated its default model to GPT-5.6+, indicating a move towards more advanced language models in AI tooling. Concurrently, Transformer Reinforcement Learning (TRL) has demonstrated training capabilities e…
-
Generative AI testing ensures accuracy, safety, and fairness of AI outputs
Generative AI testing is crucial for ensuring the accuracy, safety, and fairness of AI outputs, as these models can produce errors or harmful content. The process involves defining test cases, running AI models, and com…
-
Top 5 LLM Evaluation Frameworks for Release Engineering Ranked
A recent analysis highlights Promptfoo as the leading LLM evaluation framework for release engineering, particularly for its CI/CD integration that can block builds on failed tests. DeepEval is recommended for Python-ba…
-
Microsoft Archives PyRIT LLM Red-Teaming Tool; Alternatives Emerge
Microsoft has archived its open-source LLM red-teaming framework, PyRIT, on GitHub as of March 27, 2026. This means the tool is no longer receiving updates, commits, or issue triage, making it a less reliable foundation…
-
Open-source tools for LLM prompt injection testing compared
Several open-source tools exist to test LLM applications for prompt injection vulnerabilities, but they are not interchangeable and cater to different testing needs. Promptfoo, Giskard, and sentinel-scan-cli focus on ap…
-
New tool evalmut tests LLM evaluation suites with intentionally broken models
A new tool called evalmut has been developed to address the limitations of LLM evaluation suites by introducing mutation testing. The tool features a "reference-fleet" of six deterministic models, each intentionally bro…
-
Z.ai's GLM-5.3 powers dev.to's content pipeline with improved instruction following
Z.ai has released its latest model, GLM-5.3, which is now powering the daily content pipeline for dev.to. The new model, accessible via an OpenAI-compatible API, has demonstrated tighter instruction following and improv…
-
Promptfoo offers eval-driven prompt development
Promptfoo is a tool designed to improve the development of AI prompts by replacing subjective
-
Agent system prompts may harm LLM performance, study finds
An experiment measuring the effectiveness of agent system prompts found that most prompts actually degrade model performance. Boris Cherny of Anthropic noted that Claude Code became more intelligent after 80% of its sys…
-
Bonsai 27B 2-bit model shows promise for local use but lags in complex tasks
A recent comparison evaluated the Bonsai 27B 2-bit model against other local LLMs like Qwen3 14B, GPT OSS 20B, and Gemma 4-12B on a MacBook M1. Bonsai 27B performed well on shorter tasks, successfully completing nine ou…
-
EvalPort introduces 11 grader types for flexible LLM evaluation
EvalPort has developed a flexible grader system designed to accommodate various LLM evaluation frameworks. The system features 11 distinct grader types, each with specific parameters and evaluation methods, aiming for b…
-
Local classifier replaces costly LLM-as-a-Judge for AI evaluations
An alternative to using large language models (LLMs) for evaluation has been developed, addressing the high costs and latency associated with API-based judging. This new method employs a local binary classifier, trained…
-
OpenAI's Promptfoo Acquisition Sparks Debate on LLM Evaluation Independence
The acquisition of Promptfoo by OpenAI has prompted a re-evaluation of LLM evaluation tools, highlighting concerns about vendor dependency and cost. The author proposes an alternative approach using a custom-trained cla…
-
LLM prompt edits bypass testing, causing significant accuracy drops
A significant drop in LLM extraction accuracy, from 0.87 to 0.78, occurred after a minor one-word edit to the system prompt. This highlights a critical gap in current LLM application development, where prompt changes of…
-
New AI Crash Test tool offers auditable LLM vulnerability grading
A new browser-based tool called The AI Crash Test offers a deterministic method for evaluating LLM vulnerabilities, avoiding the use of LLM judges to ensure auditable results. The tool allows users to test models direct…
-
New tool 'muteval' tests LLM evaluation robustness
Ashwin Ugale has developed a new tool called muteval, inspired by mutation testing in software engineering, to evaluate the robustness of Large Language Model (LLM) evaluation suites. Muteval deliberately degrades a sys…
-
LLM judges introduce systematic biases, skewing evaluations
Using Large Language Models (LLMs) as judges for evaluating other LLM outputs introduces systematic biases, such as position, verbosity, and self-preference, which cannot be averaged out like random noise. These biases …
-
LLM-as-judge CI gates incur unexpected costs; deterministic alternatives offer savings
An engineer discovered that using LLM-as-judge metrics for CI/CD evaluation gates incurs significant, ongoing costs. These gates, which assess pull requests, can generate substantial bills due to repeated API calls to m…