langsmith
PulseAugur coverage of langsmith — every cluster mentioning langsmith across labs, papers, and developer communities, ranked by signal.
- competes with Langfuse 70%
- competes with Helicone 70%
- competes with Braintrust Ai 70%
- competes with Arize Phoenix 70%
- used by Promptfoo 70%
- used by vcrpy 70%
- competes with Phoenix 70%
- used by langchain-core 70%
- affiliated with langchain-core 70%
- competes with Arize 70%
- competes with Future AGI 70%
- used by aiohttp 70%
- 2026-05-28 product_launch AWS and LangChain released a guide detailing how to use LangSmith on AWS for evaluating AI agents. source
13 day(s) with sentiment data
-
LangChain releases v1.3.15 with middleware and model integration updates
LangChain has released version 1.3.15, introducing several new features and bug fixes to its development framework. Key updates include enhancements to middleware components like AgentMiddleware, SummarizationMiddleware…
-
LLM observability platforms diverge on advanced features as market booms
The LLM observability and evaluation platform market is rapidly expanding, with projections reaching $9.26 billion by 2030. Platforms are diversifying into AI-native tools, open-source evaluation libraries, AI gateways,…
-
LLM observability tools capture traces but limit assertion granularity
Observability tools for LLM agents, such as Langfuse, LangSmith, and Phoenix, offer ways to capture production traces, but their default configurations for defining inputs and assertions can be limiting. The author argu…
-
RAG observability tools like LangSmith and Arize Phoenix gain traction
Retrieval-Augmented Generation (RAG) systems are moving from experimental stages to critical production components for chatbots and other applications. To ensure these systems function effectively, robust observability …
-
OpenAI's Promptfoo Acquisition Sparks Debate on LLM Evaluation Independence
The acquisition of Promptfoo by OpenAI has prompted a re-evaluation of LLM evaluation tools, highlighting concerns about vendor dependency and cost. The author proposes an alternative approach using a custom-trained cla…
-
New tool converts agent failures into fine-tuning data
A new open-source tool called trace2train has been released to convert failed agent traces into supervised fine-tuning (SFT) or Direct Preference Optimization (DPO) training data. Developed as a local CLI tool, it aims …
-
LangSmith LLM Gateway adds runtime spend limits and PII redaction
LangSmith's new LLM Gateway offers runtime governance for AI agents, addressing budget and compliance risks. It integrates directly into the request path between agents and LLM providers, enabling features like hard spe…
-
LangChain Ecosystem Explained: Building Blocks vs. Operational Tools
The LLM development ecosystem is experiencing "Lang-fatigue" due to a proliferation of tools with similar naming conventions. This guide clarifies the distinctions between open-source building blocks like LangChain, Lan…
-
LLM prompt edits bypass testing, causing significant accuracy drops
A significant drop in LLM extraction accuracy, from 0.87 to 0.78, occurred after a minor one-word edit to the system prompt. This highlights a critical gap in current LLM application development, where prompt changes of…
-
OpenSmith releases major update for local LLM tracing
OpenSmith, a local-first alternative to LangSmith for tracing LLM pipelines, has released a significant update. The new version features a redesigned dashboard with real-time updates, enhanced search and filtering capab…
-
Developer shares 20-minute LLM agent model swap testing method
A developer has shared a 20-minute process for testing LLM agent model swaps to prevent subtle behavioral regressions. The method involves recording baseline agent behavior with specific scenarios using the `whatbroke-c…
-
Agentic AI Orchestration powers real-time fraud detection system
This article details the architecture of a production-ready agentic fraud detection system, focusing on the integration of multiple AI agents and a human-in-the-loop workflow. It emphasizes moving beyond simple model pe…
-
LangChain updates Anthropic integration, adds Claude Opus 4.8 structured output
LangChain has released updates for its Anthropic integration, with versions 1.5.1 and 1.5.0 introducing several enhancements. Version 1.5.1 specifically enables structured output for Claude Opus 4.8 and supports LangSmi…
-
LangChain updates Fireworks integration to v1.5.1, adds LangSmith gateway support
LangChain has released version 1.5.1 of its langchain-fireworks integration, following closely on the heels of version 1.5.0. This latest update introduces support for the LangSmith gateway via environment variables, be…
-
LangChain releases updates across multiple libraries, adding new features and fixes
LangChain has released several updates across its core libraries, including langchain-openai, langchain-core, langchain-openrouter, and langchain-xai. These updates introduce new features such as support for the LangSmi…
-
Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics · 8 sources tracked
This series of articles details the creation of production-grade evaluation pipelines for Large Language Models (LLMs), moving beyond subjective "vibe checks" to implement automated metrics. The authors emphasize the ne…
-
Developer builds RAG app using LangChain, OpenAI, and Pinecone
This article details the construction of a basic Retrieval-Augmented Generation (RAG) application using Python. The RAG system enhances large language models by allowing them to access and utilize information from priva…
-
AI Agents: Users Seek Robust Testing and Evaluation Methods
A user on the r/LocalLLaMA subreddit is seeking advice on how to reliably test and evaluate AI agents, expressing frustration with the difficulty of ensuring their agents function correctly beyond manual, "vibe-based" c…
-
Scaling RAG to 10 Million Documents Requires Advanced Ingestion and Retrieval Techniques
Scaling Retrieval-Augmented Generation (RAG) systems from a few thousand documents to millions presents significant challenges that often break simpler implementations. Production-scale RAG requires robust ingestion pip…
-
Harbor adds LangSmith integration for swappable AI agent evaluation backends
Harbor, an open-source framework for evaluating AI agents, has integrated LangSmith's production sandboxes. This allows users to write evaluation code once and run it across various environments, including Daytona, E2B,…