LLM observability tools capture traces but limit assertion granularity

Observability tools for LLM agents, such as Langfuse, LangSmith, and Phoenix, offer ways to capture production traces, but their default configurations for defining inputs and assertions can be limiting. The author argues that capturing an agent's failure requires defining the input as the state at the point of failure, not just the initial user message, and that assertion slots often capture a single output rather than the full trajectory of tool calls. DeepEval is highlighted as an exception, providing schemas that can directly model assertions about tool call sequences. AI

IMPACT Highlights limitations in current LLM agent testing tools, suggesting a need for more granular assertion capabilities.

RANK_REASON The item discusses specific software tools for LLM observability and their features.

Read on dev.to — LLM tag →

• DatasetItem
• DeepEval
• Future AGI
• Langfuse
• LangSmith
• LLM_INPUT_MESSAGES
• LLM_TOOLS
• ObservationDetailViewHeader.tsx
• Phoenix

• product
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →