Promptfoo
PulseAugur coverage of Promptfoo — every cluster mentioning Promptfoo across labs, papers, and developer communities, ranked by signal.
- 2026-05-20 product_launch Promptfoo integrates its attack plugins with the OWASP LLM Top 10 2025 security categories. source
8 day(s) with sentiment data
-
OpenAI Evals service shutting down; users urged to migrate
OpenAI is discontinuing its Evals service, with read-only access ending October 31st and full inaccessibility by November 30th. The Agent Builder and v1/prompts API will also be discontinued on the latter date. Users ar…
-
AI evaluation firm Authensor shifts focus to auditing AI graders, fixing bugs
Authensor, an AI evaluation integrity firm, has retired its SafeClaw and Authensor policy engine products to focus on auditing AI grading tools. The company has identified and fixed numerous bugs across various AI evalu…
-
AI graders fail to distinguish spaces, impacting code evaluations
A coding evaluation issue has been identified where AI models and grading tools struggle to differentiate between standard spaces and non-breaking spaces (U+00A0). This problem was discovered by Driftproof when their ow…
-
Claude Code evaluation metrics detailed for AI agent consistency
This article details three key metrics for evaluating AI agents, specifically focusing on Claude Code. The metrics are pass@k, which checks if an agent can solve a task within a set number of attempts; pass^k, measuring…
-
9 LLM evaluation tools for production teams detailed
A recent engineering shortlist highlights nine tools designed to evaluate Large Language Models (LLMs) in production environments. These tools, including DeepEval, Ragas, Promptfoo, MLflow, Langfuse, Arize Phoenix, TruL…
-
OpenAI to deprecate hosted Evals platform by November 2026
OpenAI is deprecating its hosted Evals platform, with existing evaluations becoming read-only on October 31, 2026, and the dashboard and API shutting down entirely on November 30, 2026. Users are advised to export their…
-
AI agents fail security test, approving fraudulent $75K invoice
A Capture The Flag (CTF) event highlighted vulnerabilities in AI agent security, where an invoice agent approved a $75,000 invoice despite a $50,000 policy limit, due to a claim of CFO pre-approval. This failure occurre…
-
LLM app testing: CI-integrated evals with LLM judges
Testing large language model applications requires a different approach than traditional software due to their non-deterministic nature. A recommended strategy involves creating a set of approximately 50 real-world inpu…
-
6 MCP Evaluation Tools for Server Developers Compared
This article reviews six tools designed for evaluating the performance of agents interacting with MCP (Message, Command, and Protocol) servers. The evaluation focuses on an agent's ability to correctly select tools, pas…
-
Promptfoo vs. DeepEval: Choosing the right LLM evaluation tool
The author compares three tools for evaluating LLM responses: Promptfoo, DeepEval, and a custom-built LLM Judge Rubric Builder. Promptfoo is recommended for existing YAML-based evaluations and CI integration, while Deep…
-
New AI agent security tools verify real-world actions, not just reports
A new category of open-source tools is emerging to address the gap between what AI agents report doing and their actual actions, particularly concerning security vulnerabilities. These tools focus on verifying real-worl…
-
New LLM-RedKit tool targets agentic prompt injection vulnerabilities
A new open-source tool called LLM-RedKit has been developed to address the evolving threat of prompt injection attacks against AI agents. Unlike previous tools focused on static prompts, LLM-RedKit specifically tests ag…
-
LLM chatbot testing checklist emphasizes property-based assertions and OWASP security
Testing LLM chatbots and RAG applications requires a shift from traditional deterministic checks to property-based assertions, focusing on required facts, refusal of out-of-scope queries, and adherence to format. Develo…
-
Langfuse, Phoenix, Promptfoo, and Iris: AI agent evaluation tools compared
Four distinct tools—Langfuse, Arize Phoenix, Promptfoo, and Iris—offer different approaches to evaluating AI agent performance. Langfuse and Arize Phoenix are observability platforms that have incorporated evaluation fe…
-
Promptfoo enables test-driven development for LLM prompts
Promptfoo is an open-source command-line tool designed to enable test-driven development for large language model prompts. It allows users to define prompts, providers, and test cases in a YAML configuration, compare di…
-
New tool muteval finds missing LLM tests by degrading systems
A new tool called muteval aims to address a critical blind spot in LLM testing by identifying tests that are missing rather than just reporting on existing ones that fail. Unlike traditional test runners, muteval intent…
-
TypeScript experiment compares agent trajectory checking tools
This item is a brief technical note discussing a small experiment in TypeScript. It compares two tools, AgentInspect and Promptfoo, for performing local checks on agent trajectories. The experiment focuses on the coding…
-
Promptfoo upgrades to GPT-5.6+, TRL trains beyond 1M tokens
Promptfoo has updated its default model to GPT-5.6+, indicating a move towards more advanced language models in AI tooling. Concurrently, Transformer Reinforcement Learning (TRL) has demonstrated training capabilities e…
-
Generative AI testing ensures accuracy, safety, and fairness of AI outputs
Generative AI testing is crucial for ensuring the accuracy, safety, and fairness of AI outputs, as these models can produce errors or harmful content. The process involves defining test cases, running AI models, and com…
-
Top 5 LLM Evaluation Frameworks for Release Engineering Ranked
A recent analysis highlights Promptfoo as the leading LLM evaluation framework for release engineering, particularly for its CI/CD integration that can block builds on failed tests. DeepEval is recommended for Python-ba…