langsmith
PulseAugur coverage of langsmith — every cluster mentioning langsmith across labs, papers, and developer communities, ranked by signal.
- 2026-05-28 product_launch AWS and LangChain released a guide detailing how to use LangSmith on AWS for evaluating AI agents. source
12 day(s) with sentiment data
-
New tool AgentBreak finds LLM email agents vulnerable to inbox hijacking
A security vulnerability has been identified in LLM-based email agents that utilize tools, specifically through indirect prompt injection. An attacker can craft an email that manipulates the agent into forwarding its en…
-
LangChain updates fireworks integration with dependency fixes
LangChain has released version 1.4.3 of its langchain-fireworks integration, which includes several dependency updates and minor improvements. The release addresses compatibility issues by capping aiohttp below version …
-
EU data rules force novel LLM eval approach for meeting assistant
A developer building a meeting assistant faced challenges with real-time production evaluations due to strict EU data residency rules. Standard online evaluation methods, which require access to inputs and outputs, were…
-
LangChain updates OpenRouter integration with new features and dependency bumps
LangChain has released version 0.2.4 of its OpenRouter integration, updating dependencies and introducing new features. This release bumps the OpenRouter library version to 0.9.2, drops a workaround for file handling, a…
-
Runcap introduces pre-flight cost control for AI agents
A new tool called Runcap has emerged to address the issue of runaway AI agent costs, offering a unique pre-flight cost control mechanism. Unlike observability tools like Langfuse or gateways such as LiteLLM, Runcap oper…
-
Developer builds PII firewall to block sensitive data from LLM prompts
A developer built a PII firewall for LLM interactions to prevent sensitive data from being sent to cloud-based models. The system, implemented using FastAPI and Microsoft Presidio, scans prompts before they reach models…
-
LLM observability tools miss critical audio layer for voice agents
Observability tools for LLMs primarily focus on tracing model calls, including prompts, completions, and latency, which is insufficient for voice agents. Failures in voice agents often occur in the audio layer, such as …
-
LangChain releases updates for core libraries and partner integrations
LangChain has released several updates across its core libraries and partner integrations. Version 1.3.11 of the main LangChain library includes fixes for OpenAI-compatible models and dependency updates. The `langchain-…
-
AI Agents Require Software-Like Version Control for Stability
The article discusses the critical need for version control in AI agents, likening their configuration complexity to software code. It highlights the risks of deploying changes directly to production without proper test…
-
LangChain simplifies LLM app development with standardized components
LangChain is a framework designed to simplify the development of LLM applications by providing a standardized interface for various components. It abstracts away the complexities of interacting with different AI models,…
-
LangSmith enables LLM regression testing and audit trails
This two-part series explores LLM observability and traceability, focusing on the LangSmith platform. Part 1 details how to make LLM applications replayable and create tamper-evident audit logs using LangSmith's tracing…
-
LLM Eval Tooling: Key Questions for Long-Term Usability
Choosing LLM evaluation tooling requires careful consideration beyond just features, as vendor lock-in can become a significant issue. The article advises asking four key questions before committing to a tool, focusing …
-
Voice agent testing fails on rare inputs; simulation is key
Testing voice agents with real call transcripts can create a false sense of security, as it fails to capture rare or novel user behaviors. A developer experienced a critical failure when a caller switched languages mid-…
-
LLM Observability Tools Map: LangSmith, Langfuse, Braintrust Emerge
The LLM observability landscape is evolving, with several tools emerging to address the need for monitoring and understanding LLM applications. Key platforms like LangSmith, Langfuse, Braintrust, Helicone, and Arize Pho…
-
Developer releases Regtrace CLI for detecting silent LLM regressions
A developer has created Regtrace, an open-source command-line tool designed to catch silent regressions in large language models. Unlike traditional testing methods, Regtrace focuses on detecting subtle errors introduce…
-
Student builds circuit breaker to stop runaway LLM agent costs
A student developer created AgentBrake, a Python decorator to prevent LLM agents from incurring excessive costs or performing unintended actions. The tool acts as a circuit breaker, monitoring tool calls for infinite lo…
-
LLM invoices surge due to unmonitored agent loops and missing outcome assertions
LLM invoices are increasing significantly due to token consumption per task, not just per-token pricing. Observability platforms often fail to identify cost leaks because they don't track the actual business outcome of …
-
LangChain releases updates for core libraries and perplexity integration
LangChain has released several updates across its core libraries, including `langchain-perplexity`, `langchain-core`, and the main `langchain` package. These updates focus on bug fixes, dependency upgrades, and minor fe…
-
AWS and LangChain detail AI agent evaluation framework
AWS and LangChain have collaborated on a guide for evaluating AI agents, leveraging LangSmith on AWS. The guide details methods for testing agent behavior, including offline evaluations with pytest and online monitoring…
-
LLM proxy gains live dashboard for zero-instrumentation monitoring
A developer has enhanced their LLM proxy tool, Trooper, to include a live dashboard for real-time monitoring of agent interactions. This dashboard, accessible via a URL change, automatically extracts and displays sessio…