langsmith
PulseAugur coverage of langsmith — every cluster mentioning langsmith across labs, papers, and developer communities, ranked by signal.
- developed by langchain-openai 90%
- acquired by Promptfoo 90%
- partners with Fireworks 90%
- used by Promptfoo 70%
- competes with DeepEval 70%
- competes with Braintrust Ai 70%
- competes with Helicone 70%
- competes with Phoenix 70%
- competes with Arize Phoenix 70%
- used by vcrpy 70%
- competes with Promptfoo 70%
- affiliated with langchain-core 70%
- 2026-08-25 product_launch Fireworks AI and LangChain are co-hosting a workshop on August 25th focused on building custom evaluation models for LangSmith. source
- 2026-08-25 product_launch Fireworks and LangChain are co-hosting a workshop to demonstrate building custom evaluation models for LangSmith. source
- 2026-05-28 product_launch AWS and LangChain released a guide detailing how to use LangSmith on AWS for evaluating AI agents. source
9 day(s) with sentiment data
-
Developer finds LLMs perform best with limited tasks, code handles complexity
A developer building a survey-analysis agent using LangChain and Python discovered that language models should be given minimal tasks, with ordinary code handling complex operations. The agent initially relied on a sing…
-
New LLM workflows and tracing tools aid in debugging complex AI applications
Developers are exploring new methods to debug and verify Large Language Model (LLM) applications. One approach involves creating a repeatable workflow where an LLM proposes potential bugs, and deterministic tools like c…
-
AI agents and LLMs: practical applications and management strategies
Several articles discuss the practical application and management of AI agents and large language models (LLMs). One piece explores the use of assertion-free tests and other techniques for managing code changes, while a…
-
Catching silent LLM failures with AI observability and evaluation
This tutorial demonstrates how to implement AI observability and evaluation to catch silent failures in LLM applications. It outlines a four-component architecture involving instrumentation, an observability backend, an…
-
Linklbee Scan tool helps users find LLM API cost leaks
NOTEGG has released Linklbee Scan, an open-source Python tool designed to help users identify cost inefficiencies in their LLM API usage. The tool analyzes existing log files on a user's machine, without requiring log u…
-
New Python library `replay.py` enables reproducible AI agent runs
A new Python library called `replay.py` has been developed to address the challenge of reproducing agent runs in AI systems. Unlike traditional transcripts that only record model outputs, `replay.py` captures detailed t…
-
Decision models offer massive cost savings over LLMs for simple AI tasks · 2 sources tracked
New discriminative decision models, like Jev and OpenAI's Decisions API, are emerging as a cost-effective alternative to large language models for tasks that require simple choices rather than complex reasoning. These m…
-
pylinxi offers Pythonic durability for AI agents, bypassing graph complexity
pylinxi is a new open-source Python library designed to enhance the recoverability of AI agents by saving their progress. Unlike graph-based frameworks like LangGraph, which can introduce overhead for complex logic and …
-
Author details 90-test case RAG evaluation framework
Building a robust Retrieval-Augmented Generation (RAG) system requires a comprehensive evaluation framework to prevent silent failures like hallucinations and irrelevant retrieval. The author details a 90-test case appr…
-
Observability stack for Node.js AI apps detailed
A developer has outlined an observability stack for Node.js AI applications, emphasizing the importance of monitoring before launch. The proposed stack includes tools for standard tracing, logging, metrics, error tracki…
-
AI Agent Learns Terminology with Custom BLEU Score
The author details the creation of a small AI agent designed to learn and utilize specific terminology. This agent was developed using tools like LangChain and Python, and it was trained to evaluate its own performance …
-
6 MCP Evaluation Tools for Server Developers Compared
This article reviews six tools designed for evaluating the performance of agents interacting with MCP (Message, Command, and Protocol) servers. The evaluation focuses on an agent's ability to correctly select tools, pas…
-
Top 5 Prompt Management Tools for LLMs in 2026 Reviewed
The article reviews five prompt management tools for LLMs, highlighting Braintrust Ai as a strong default for teams needing to tie prompt changes to experiments and production traces. Other tools like PromptLayer, LangS…
-
Langfuse, Phoenix, LangSmith: Choosing LLM Observability Tools
This article compares three popular tools for LLM observability: Langfuse, Phoenix, and LangSmith. It delves into aspects such as licensing models, self-hosting capabilities, integration with CI/CD pipelines for evaluat…
-
LangChain updates Hugging Face integration with dependency bumps
LangChain has released version 1.2.3 of its langchain-huggingface library, introducing several dependency updates and minor fixes. The release includes bumps to various libraries such as LangGraph SDK, fsspec, sentence-…
-
7 production patterns for training AI agents detailed
Training an AI agent for production involves a cyclical process focused on infrastructure and data. Key steps include instrumenting every agent run to capture detailed traces, using these traces to build evaluation data…
-
AI agent frameworks audited for cost visibility, revealing significant gaps
A recent audit of five popular AI agent frameworks revealed significant gaps in cost visibility and control. While frameworks like LangGraph with LangSmith and Amazon Bedrock AgentCore offer robust token-to-dollar conve…
-
LLM routers cause silent quality collapse, eroding user trust over 6 weeks
LLM routers, while offering significant cost savings, can lead to a silent collapse in response quality that goes undetected for weeks. This degradation, often caused by miscalibrated classifiers or semantic drift in ch…
-
LangSmith vs. MLflow: Comparing RAG Application Observability Tools
This article compares LangSmith and MLflow, two platforms used for observing and tracing operations within retrieval-augmented generation (RAG) applications. It highlights how both tools can provide insights into the sa…
-
Newsletter Issue 108 Features ShinyReact, LangSmith, and RAG Cookbook
Issue 108 of a newsletter highlights ShinyReact as the open-source project of the week. It also points to new learning resources, including LangSmith, specialist agents, and software factories. Additionally, the issue f…