Skip to content
Prepline
LibraryAI Agents & SDKs46 min readUpdated 2026-06-13
Comprehensive Technical Course

Agent Tracing & Observability

From Black Box to Glass Box: Making Your AI Agents Observable

Author: Mustafa Furniturewala Reading Time: ~2.5 hours Level: VP / Staff+ Engineering Updated: June 2026

You're running agents in production. Maybe it's a Cowork-style setup with 30+ scheduled tasks, a command dispatcher, and multi-model routing. Maybe it's a customer-facing assistant that makes tool calls you can't see. Either way, your agents are black boxes, and that's terrifying. This course takes you from zero observability to a production-grade tracing stack, with real Python code you can deploy today. No hand-waving. No vendor marketing. Just engineering.

1 Why Agent Tracing Matters

The Black Box Problem

You built an agent. It uses Claude, maybe GPT-4o for some tasks, Haiku for fast classification. It has twelve tools: file search, calendar management, email composition, web scraping, code execution, database queries. It works great in your demo. Then it goes to production.

Three days later, a user reports that the agent "did something weird." They can't be more specific than that. You check the logs. There are logs, of course, because you're not a monster. But the logs look like this:

output.log
2026-06-04 14:23:01 INFO Agent started for user_id=u_8821 2026-06-04 14:23:01 INFO Sending message to Claude... 2026-06-04 14:23:03 INFO Received response. Tokens: 1847 2026-06-04 14:23:03 INFO Tool call: search_files 2026-06-04 14:23:04 INFO Tool result received 2026-06-04 14:23:04 INFO Sending message to Claude... 2026-06-04 14:23:06 INFO Received response. Tokens: 923 ... (41 more entries like this) ... 2026-06-04 14:24:12 ERROR Agent failed: KeyError: 'content'

What happened? The agent made 47 tool calls. It failed on number 38. Your logs tell you that it happened but not why. You can see it called search_files and send_email and calendar_create, but you can't see what it passed to those tools, what it got back, what its reasoning was between steps, or why it decided to call tool #38 at all.

This is the black box problem, and print() debugging doesn't solve it. Not at scale. Not when your agent runs 500 times a day across 30+ scheduled tasks.

A Horror Story: Debugging Without Traces

Consider a real scenario. You have a Cowork-style system: a command dispatcher that routes incoming requests to specialized agents, 30+ scheduled tasks running on cron, a learning system (call it Learn Imagine v3) that adapts based on user feedback. One morning, you notice your API bill spiked 4x. Something is burning tokens.

Without tracing, here's your debugging process:

  1. Check the billing dashboard. You see $47 spent yesterday instead of the usual $12. The breakdown shows Claude Opus consumed 2.1M input tokens. But from which queries?
  2. Grep through application logs. You find 1,200 agent invocations. Some ran for 2 seconds, some for 45 seconds. You can't tell which ones were expensive because your logs don't capture token counts per turn.
  3. Add print(f"tokens: {response.usage}") everywhere. Redeploy. Wait for it to happen again.
  4. Next day: you find the culprit. Your scheduled task "daily_report_generator" got stuck in a retry loop. It called the search tool, got an empty result, asked Claude to try again, got an empty result, asked Claude to try again... 34 times. Each retry included the full conversation history, which grew with every loop. The final turn sent 89,000 input tokens.
  5. Total debugging time: two days and $47 in wasted API spend.

With proper tracing, this would have taken five minutes. You'd open your trace dashboard, sort by cost, see the outlier, click into it, and see the expanding conversation history and the retry loop in a visual waterfall. You'd fix it before lunch.

Logging vs. Tracing: They're Not the Same Thing

Engineers often conflate logging and tracing. They're fundamentally different paradigms:

Logging is event-oriented. Each log line is an independent record: "this thing happened at this time." Logs are great for auditing, error reporting, and simple debugging. But logs are flat. They don't capture relationships between events.

Tracing is relationship-oriented. A trace captures an entire operation as a tree of spans, each with a parent, a duration, metadata, and input/output data. Tracing tells you not just what happened, but why it happened in that order, how long each step took, and how data flowed from one step to the next.

For an LLM agent, the difference is stark. A log line says "called search_files." A trace span says "called search_files with query='Q2 revenue report', received 3 results in 340ms, parent span was 'Claude tool_use turn #4', which was part of trace 'daily_report_generator run #891', total trace cost so far: $0.08."

Why Traditional APM Tools Fall Short

You might think: "We already have Datadog / New Relic / Honeycomb. Can't we just use those?" Partially, but they miss critical dimensions of LLM agent behavior:

  • Token economics: Traditional APM tracks request latency and error rates. It doesn't understand that a 2-second LLM call that used 50K tokens is fundamentally different from a 2-second call that used 500 tokens, even though they look identical from an HTTP perspective.
  • Prompt/completion capture: APM tools don't know to capture the full prompt and completion text. Without this, you can't debug why an agent made a bad decision.
  • Multi-turn reasoning: An agent conversation is a sequence of LLM calls with tool calls in between. Each turn depends on the previous one. Traditional APM sees each HTTP call as independent. You lose the chain of reasoning.
  • Tool call semantics: When your agent calls a tool, you need to see what arguments it generated, whether those arguments were valid, what the tool returned, and how the agent interpreted the result. This is a domain-specific concern that generic APM doesn't model.
  • Cost attribution: You need to attribute cost to features, users, and conversations, not just to endpoints. "The /api/chat endpoint costs $2,400/month" is less useful than "the research_mode feature costs $1,800/month because it averages 12 tool calls per query."

Traditional APM is necessary but not sufficient. You need it for infrastructure health. You need agent-specific tracing for agent health.

The Cost of Unobserved Agents

Let me be concrete about what you're losing without tracing:

  • Wasted tokens: Agents that retry unnecessarily, include redundant context, or get stuck in loops. In systems with 30+ scheduled tasks, even one misbehaving task can 10x your daily spend.
  • Silent failures: The agent returns a plausible-sounding but wrong answer. Without tracing the reasoning chain, you'll never know. Your users will just quietly lose trust.
  • Hallucinated tool calls: The agent invents tool parameters that happen to match the schema but are semantically wrong. It searches for "Q2 report" when the user asked about "Q3." The tool returns results, the agent uses them, and nobody notices the slip.
  • Latency mysteries: A query takes 30 seconds. Is it the LLM? The tool? The network? Without a trace waterfall, you're guessing.
  • Impossible optimization: You can't optimize what you can't measure. Want to know if Haiku could handle 60% of your traffic? Without per-query traces showing complexity, you're flying blind.
Key Takeaway

Agent tracing is not a nice-to-have observability feature. It's the difference between running a production system and running a prayer. If your agents make tool calls, handle user data, or cost real money, you need traces. Period.

2 The Trace Data Model

Spans, Traces, and Events

Before you instrument anything, you need to understand the data model. Tracing has precise terminology, and using it correctly will save you hours of confusion when you're reading documentation for any tracing tool.

A trace represents a single end-to-end operation. In agent terms, one trace equals one user query from the moment it arrives to the moment the final response is returned. A trace has a unique trace_id and contains one or more spans.

A span represents a single unit of work within a trace. Spans have a start time, an end time, a name, and optional metadata. Critically, spans have parent-child relationships. The root span is the top-level operation. Child spans represent sub-operations. This gives you a tree structure.

An event (sometimes called a "log" in OpenTelemetry terminology) is a timestamped annotation attached to a span. Events don't have duration; they mark a point in time. "Tool call started," "Cache hit," "Retry initiated" are events.

Here's how an agent query decomposes into spans:

Trace: trace_7f3a2b (user query: "Summarize my Q2 revenue report") | +-- [ROOT SPAN] agent_run (duration: 8.2s, cost: $0.034) | +-- [SPAN] llm_call_1 (Claude Sonnet, 1.8s) | | input_tokens: 2,100 output_tokens: 340 | | tool_use: search_documents | +-- [EVENT] tool_selected: search_documents | +-- [SPAN] tool_exec: search_documents (0.9s) | | args: {query: "Q2 revenue report", limit: 5} | | result: 3 documents found | +-- [EVENT] db_query (0.7s) | +-- [EVENT] result_filtered (0.1s) | +-- [SPAN] llm_call_2 (Claude Sonnet, 3.1s) | | input_tokens: 8,400 output_tokens: 1,200 | | tool_use: read_document | +-- [EVENT] tool_selected: read_document | +-- [SPAN] tool_exec: read_document (0.4s) | | args: {doc_id: "rev_q2_2026", pages: "1-5"} | | result: 4,200 chars extracted | +-- [SPAN] llm_call_3 (Claude Sonnet, 2.0s) | input_tokens: 12,800 output_tokens: 890 | finish_reason: end_turn +-- [EVENT] final_response_generated

Notice the structure. The root span encompasses the entire operation. Each LLM call and tool execution is a child span. Events are annotations within spans. You can see the data flow: the first LLM call decided to search, the tool returned results, the second LLM call decided to read a specific document, the tool returned content, and the third LLM call generated the summary.

The Span Data Model in Python

Let's formalize this. Here's a Python dataclass that models a span suitable for LLM agent tracing:

models.py
from dataclasses import dataclass, field from typing import Optional, Any from datetime import datetime from enum import Enum import uuid class SpanKind(Enum): AGENT_RUN = "agent_run" LLM_CALL = "llm_call" TOOL_EXEC = "tool_execution" RETRIEVAL = "retrieval" GUARDRAIL = "guardrail_check" @dataclass class SpanEvent: name: str timestamp: datetime = field(default_factory=datetime.utcnow) attributes: dict[str, Any] = field(default_factory=dict) @dataclass class Span: # Identity span_id: str = field(default_factory=lambda: uuid.uuid4().hex[:16]) trace_id: str = "" parent_span_id: Optional[str] = None name: str = "" kind: SpanKind = SpanKind.AGENT_RUN # Timing start_time: datetime = field(default_factory=datetime.utcnow) end_time: Optional[datetime] = None # LLM-specific metadata model: Optional[str] = None input_tokens: int = 0 output_tokens: int = 0 cost_usd: float = 0.0 temperature: Optional[float] = None # Input/Output capture input_data: Optional[Any] = None # prompt or tool args output_data: Optional[Any] = None # completion or tool result # Tool-specific tool_name: Optional[str] = None tool_args: Optional[dict] = None tool_result: Optional[Any] = None # Status status: str = "ok" # ok, error, timeout error_message: Optional[str] = None # Events and children events: list[SpanEvent] = field(default_factory=list) children: list["Span"] = field(default_factory=list) def duration_ms(self) -> Optional[float]: if self.end_time and self.start_time: return (self.end_time - self.start_time).total_seconds() * 1000 return None def total_cost(self) -> float: return self.cost_usd + sum(c.total_cost() for c in self.children)

Standard Metadata Fields

Every tracing tool uses slightly different field names, but the following metadata should be captured on every LLM span. This table serves as your instrumentation checklist:

FieldTypeDescriptionWhy It Matters
trace_idstringUnique ID for the entire operationLinks all spans in one query together
span_idstringUnique ID for this spanIdentifies this specific unit of work
parent_span_idstring?ID of the parent spanBuilds the trace tree
modelstringModel used (claude-sonnet-4-20250514, gpt-4o, etc.)Cost calculation, performance comparison
input_tokensintTokens in the promptCost tracking, context window monitoring
output_tokensintTokens in the completionCost tracking, verbosity analysis
latency_msfloatWall-clock time for this spanPerformance optimization, SLO monitoring
temperaturefloat?Sampling temperature usedDebugging non-deterministic behavior
tool_namestring?Name of tool calledTool usage analytics, error attribution
tool_argsdict?Arguments passed to the toolDebugging hallucinated parameters
tool_resultany?Result returned by the toolDebugging incorrect tool behavior
finish_reasonstringWhy the LLM stopped (end_turn, tool_use, max_tokens)Detecting truncated responses
cache_hitbool?Whether prompt caching was usedCost optimization verification
errorstring?Error message if span failedFailure debugging
user_idstring?User who initiated the tracePer-user cost and error attribution
session_idstring?Conversation session IDMulti-turn conversation tracking

Trace ID Propagation in Multi-Agent Systems

In simple setups, one agent handles one query. In production systems, you often have orchestrator agents that delegate to specialist agents. A command dispatcher might route a request to a "research agent," which calls a "summarization agent," which calls a "citation verification agent." Each agent might run in a different process or even a different service.

Trace ID propagation ensures that all spans from all agents in a single user query share the same trace_id. Without this, you see five disconnected traces instead of one coherent tree.

The standard approach, borrowed from OpenTelemetry, is to pass trace context via headers or function arguments:

trace_context.py
import contextvars # Thread-local trace context using contextvars _current_trace_id: contextvars.ContextVar[str] = contextvars.ContextVar("trace_id") _current_span_id: contextvars.ContextVar[str] = contextvars.ContextVar("span_id") def get_trace_context() -> dict: """Get current trace context for propagation.""" return { "trace_id": _current_trace_id.get(None), "parent_span_id": _current_span_id.get(None), } def set_trace_context(trace_id: str, span_id: str): """Inject trace context (e.g., when receiving from another service).""" _current_trace_id.set(trace_id) _current_span_id.set(span_id)

How OpenTelemetry Maps to LLM Agent Concepts

OpenTelemetry (OTel) is the industry standard for distributed tracing. Its data model maps cleanly to LLM agents, but requires some semantic conventions that aren't yet standardized (there is an active working group for "GenAI semantic conventions"):

  • OTel Trace = Agent conversation or task execution
  • OTel Span = LLM call, tool execution, or agent reasoning step
  • OTel SpanKind.CLIENT = Outgoing LLM API call
  • OTel SpanKind.INTERNAL = Agent's decision-making logic
  • OTel Attributes = Model name, token counts, cost, tool metadata
  • OTel Events = Tool selection, cache hits, guardrail checks
  • OTel Links = References between related traces (e.g., a follow-up query linked to the original)

The emerging convention uses attribute names like gen_ai.system (e.g., "anthropic"), gen_ai.request.model, gen_ai.usage.input_tokens, and gen_ai.usage.output_tokens. If you adopt these now, your traces will be compatible with any OTel-based backend.

Design Decision: Capture Everything, Filter Later

When in doubt, capture more metadata rather than less. Storage is cheap; missing data during an incident is expensive. Capture the full prompt and completion text (with PII redaction if needed), all tool arguments, all tool results. You can always add filtering and retention policies later. You can't retroactively add data you didn't collect.

3 Tools Landscape (Comparison)

The Observability Tools Ecosystem in 2026

The LLM observability space has matured rapidly. In 2023, you had LangSmith and not much else. In 2026, there are credible options across the spectrum from open-source self-hosted to fully managed SaaS. Choosing the right tool depends on your team size, budget, compliance requirements, and existing infrastructure.

I'm going to give you my honest, opinionated assessment of each major tool. I've used most of these in production or evaluated them seriously. Where I haven't, I'll say so.

LangSmith (by LangChain)

LangSmith is the most widely adopted LLM tracing tool, largely because of LangChain's market dominance in agent frameworks. If you're already using LangChain or LangGraph, LangSmith integrates with zero additional code. You literally import it and traces appear.

Strengths: Excellent UI for trace visualization, built-in evaluation framework, dataset management for testing, hub for sharing prompts. The trace tree view is best-in-class. Playground for iterating on prompts with trace data.

Weaknesses: Tightly coupled to the LangChain ecosystem. If you're using the Anthropic or OpenAI SDKs directly, integration requires more work. Pricing can surprise you at scale. The platform has had reliability issues under heavy load. Not open source (LangSmith is SaaS, though LangChain the framework is open source).

My take: If you're a LangChain shop, just use it. If you're not, look elsewhere.

Braintrust

Braintrust started as an evaluation platform and grew into tracing. This heritage shows in the best ways: their eval framework is exceptional, and traces integrate directly with eval scores. You can trace a production query, add it to a dataset, run evals against it, and compare model versions, all in one platform.

Strengths: Best-in-class evaluation integration. Clean SDK that works with any LLM provider. Excellent logging API that's framework-agnostic. Good proxy mode for zero-code tracing. Strong cost analytics.

Weaknesses: Smaller community than LangSmith. Some advanced features require the paid tier. The UI, while functional, isn't as polished as LangSmith's trace viewer.

My take: If evaluation is your primary concern alongside tracing, Braintrust is the best choice. The combined eval+trace workflow is unmatched.

Arize Phoenix

Phoenix is the open-source option from Arize AI. It's a standalone tool you run locally or self-host. It uses OpenTelemetry under the hood, which means your instrumentation is portable: if you outgrow Phoenix, your traces work with any OTel-compatible backend.

Strengths: Fully open source (Apache 2.0). Self-hosted, so your data never leaves your infrastructure. OTel-native, so you're not locked in. Good for compliance-sensitive environments. Free. Active community.

Weaknesses: You own the infrastructure. No managed service (though Arize's commercial platform offers one). The UI is functional but spartan compared to commercial tools. Less polished eval integration.

My take: If you need self-hosted and open source, Phoenix is the clear winner. It's also a great starting point if you're not sure what you need yet, since it's free and OTel-based.

Helicone

Helicone takes a unique approach: it's a proxy. You change your API base URL from api.anthropic.com to anthropic.helicone.ai, and every request is logged automatically. Zero code changes to your application.

Strengths: Literally zero-code setup for basic tracing. Works with any SDK or framework. Excellent cost dashboard. Good for getting started quickly. Rate limiting and caching features built in.

Weaknesses: Being a proxy adds latency (typically 10-50ms). You're routing all your LLM traffic through a third party. Limited ability to trace non-LLM operations (tool calls, custom logic). The proxy model makes it hard to capture tool execution details.

My take: Helicone is perfect for teams that want immediate visibility with zero engineering effort. It's not sufficient for deep agent debugging, but it's a great complement to a more comprehensive solution.

Weights & Biases Weave

W&B Weave brings the maturity of W&B's ML experiment tracking to LLM observability. If your team already uses W&B for model training, Weave provides a unified platform for both traditional ML and LLM ops.

Strengths: Integration with W&B's broader ML platform. Good versioning for prompts and models. Strong visualization. Enterprise features (SSO, audit logs, etc.).

Weaknesses: The LLM-specific features feel bolted onto a platform designed for traditional ML. Pricing can be opaque. The learning curve is steeper if you're not already a W&B user.

My take: Use it if you're already in the W&B ecosystem. Don't adopt W&B just for LLM tracing.

OpenTelemetry (Direct)

You can skip all vendor tools and instrument your agents with raw OpenTelemetry. Send traces to any OTel-compatible backend: Jaeger, Grafana Tempo, Honeycomb, Datadog, or your own collector. This gives you maximum flexibility and zero vendor lock-in.

Strengths: Industry standard. No vendor lock-in. Works with your existing observability stack. You control everything.

Weaknesses: You build everything yourself. No pre-built LLM trace viewer. No eval integration. No cost analytics dashboard. Significant engineering investment.

My take: This is the right choice for platform teams at scale (50+ engineers) who want to integrate LLM tracing into their existing observability stack. It's overkill for small teams.

Anthropic's Usage API

Anthropic provides usage tracking through the API response's usage field and the admin API. It's not tracing in the traditional sense, but it gives you token counts, costs, and rate limit information per request.

Strengths: Built in. No additional setup. Accurate token counts straight from the source.

Weaknesses: Not a tracing tool. No trace visualization, no span trees, no eval integration. Just per-request metrics.

My take: Always capture the usage data from API responses. Layer a real tracing tool on top.

Comprehensive Comparison

Tool Pricing Setup Key Features OSS Self-Host Best For
LangSmith Free tier, then $39+/seat/mo Low (LangChain), Medium (other) Trace UI, evals, datasets, playground No Enterprise only LangChain teams
Braintrust Free tier, usage-based after Low Evals + tracing, proxy mode, scoring No No Eval-focused teams
Arize Phoenix Free (OSS) Medium OTel-native, self-hosted, trace viewer Yes Yes Privacy-sensitive, budget-conscious
Helicone Free tier, then usage-based Very Low Proxy-based, cost dashboard, caching Yes Yes Quick wins, cost monitoring
W&B Weave Free tier, then $50+/seat/mo Medium ML platform integration, versioning No Enterprise only Existing W&B users
OpenTelemetry Free (your infra costs) High Universal standard, any backend Yes Yes Platform teams at scale
Anthropic Usage Free (included) Zero Token counts, cost per request N/A N/A Baseline metrics

Build vs. Buy: The Decision Framework

The build vs. buy decision for LLM tracing depends on three factors:

  1. Team size: Under 10 engineers? Buy. The engineering cost of building and maintaining a tracing system exceeds the subscription cost of any SaaS tool. 50+ engineers with a platform team? Build on OTel, potentially with an open-source frontend like Phoenix.
  2. Data sensitivity: If your prompts contain PII, PHI, or proprietary data, self-hosted is often a compliance requirement. This narrows your options to Phoenix, Helicone (self-hosted), or raw OTel.
  3. Existing infrastructure: If you already have Datadog or Grafana, adding OTel-based LLM tracing to your existing dashboards is often simpler than adopting a new SaaS tool. Your ops team already knows how to manage alerts, dashboards, and retention in the existing system.

Opinionated Recommendations by Team Size

Solo developer or small startup (1-5 people): Start with Helicone for instant cost visibility. Add Braintrust when you need evals. Total cost: $0-50/month.

Growth-stage team (5-20 people): Use Braintrust or LangSmith as your primary platform. Capture everything. The investment in trace infrastructure will pay back 10x when you're debugging production issues at 2 AM. Total cost: $200-500/month.

Enterprise / platform team (20+ people): Build on OpenTelemetry. Use Phoenix or a commercial OTel backend (Honeycomb, Grafana Cloud). Integrate with your existing observability stack. Build custom dashboards. This is a 2-4 week engineering project but gives you maximum control and zero vendor lock-in.

Warning: Vendor Lock-in Is Real

Whatever tool you choose, instrument your code with a thin abstraction layer. Don't scatter langsmith.trace() calls throughout your codebase. Write a Tracer interface, implement it for your chosen backend, and swap implementations later without touching application code. You will switch tools at least once. Make it easy.

4 Instrumenting Your Agents

From Theory to Code

This is the hands-on module. We're going to build real instrumentation for real agent patterns. Every code example is designed to be copied into a production codebase with minimal modification. I'll cover five patterns, from simple to complex.

Pattern 1: A Basic Trace Decorator

The simplest useful instrumentation is a decorator that wraps any LLM call, captures timing, token usage, and cost, and stores the span. This works with any LLM provider.

trace_decorator.py
import time import uuid import functools import json from datetime import datetime from typing import Callable, Any # In-memory span storage (replace with your backend) _spans: list[dict] = [] # Cost per 1K tokens (update with current pricing) MODEL_COSTS = { "claude-sonnet-4-20250514": {"input": 0.003, "output": 0.015}, "claude-haiku-4-20250414": {"input": 0.0008, "output": 0.004}, "claude-opus-4-20250514": {"input": 0.015, "output": 0.075}, "gpt-4o": {"input": 0.0025, "output": 0.01}, } def calc_cost(model: str, input_tok: int, output_tok: int) -> float: costs = MODEL_COSTS.get(model, {"input": 0, "output": 0}) return (input_tok / 1000) * costs["input"] + (output_tok / 1000) * costs["output"] def trace_llm(model: str, name: str = "llm_call"): """Decorator to trace any function that returns an LLM response.""" def decorator(func: Callable) -> Callable: @functools.wraps(func) def wrapper(*args, **kwargs) -> Any: span = { "span_id": uuid.uuid4().hex[:16], "name": name, "model": model, "start_time": datetime.utcnow().isoformat(), "input_preview": str(args)[:500], } start = time.perf_counter() try: result = func(*args, **kwargs) elapsed = (time.perf_counter() - start) * 1000 # Extract usage from Anthropic/OpenAI response usage = getattr(result, "usage", None) if usage: span["input_tokens"] = usage.input_tokens span["output_tokens"] = usage.output_tokens span["cost_usd"] = calc_cost( model, usage.input_tokens, usage.output_tokens ) span["latency_ms"] = round(elapsed, 2) span["status"] = "ok" _spans.append(span) return result except Exception as e: span["status"] = "error" span["error"] = str(e) span["latency_ms"] = round( (time.perf_counter() - start) * 1000, 2 ) _spans.append(span) raise return wrapper return decorator

Pattern 2: Tracing Claude Messages API with Tool Use

This is the pattern you'll use most often: tracing a Claude conversation that includes tool use. The key insight is that you need to trace both the LLM calls and the tool executions, and connect them with parent-child relationships.

claude_tool_tracing.py
import anthropic import time import uuid from dataclasses import dataclass, field, asdict from typing import Any @dataclass class TracedSpan: span_id: str = field(default_factory=lambda: uuid.uuid4().hex[:16]) parent_id: str | None = None name: str = "" kind: str = "generic" start: float = field(default_factory=time.perf_counter) metadata: dict = field(default_factory=dict) class TracedAgent: def __init__(self, tools: list[dict], tool_handlers: dict): self.client = anthropic.Anthropic() self.tools = tools self.tool_handlers = tool_handlers self.spans: list[dict] = [] def run(self, user_msg: str, model: str = "claude-sonnet-4-20250514") -> str: trace_id = uuid.uuid4().hex[:16] root_span = TracedSpan(name="agent_run", kind="root") messages = [{"role": "user", "content": user_msg}] turn = 0 while True: turn += 1 # --- Trace the LLM call --- llm_span = TracedSpan( name=f"llm_call_{turn}", kind="llm", parent_id=root_span.span_id, ) t0 = time.perf_counter() response = self.client.messages.create( model=model, max_tokens=4096, tools=self.tools, messages=messages, ) llm_span.metadata = { "model": model, "input_tokens": response.usage.input_tokens, "output_tokens": response.usage.output_tokens, "stop_reason": response.stop_reason, "latency_ms": round((time.perf_counter() - t0) * 1000), "trace_id": trace_id, } self.spans.append(asdict(llm_span)) # --- Handle tool use --- if response.stop_reason == "tool_use": tool_blocks = [b for b in response.content if b.type == "tool_use"] tool_results = [] for block in tool_blocks: tool_span = TracedSpan( name=f"tool:{block.name}", kind="tool", parent_id=llm_span.span_id, ) t1 = time.perf_counter() try: result = self.tool_handlers[block.name](**block.input) tool_span.metadata = { "tool": block.name, "args": block.input, "result_preview": str(result)[:500], "status": "ok", "latency_ms": round( (time.perf_counter() - t1) * 1000 ), } except Exception as e: result = f"Error: {e}" tool_span.metadata = { "tool": block.name, "args": block.input, "error": str(e), "status": "error", } self.spans.append(asdict(tool_span)) tool_results.append({ "type": "tool_result", "tool_use_id": block.id, "content": str(result), }) messages.append({"role": "assistant", "content": response.content}) messages.append({"role": "user", "content": tool_results}) else: # Final text response root_span.metadata = { "trace_id": trace_id, "total_turns": turn, "total_spans": len(self.spans), "latency_ms": round( (time.perf_counter() - root_span.start) * 1000 ), } self.spans.insert(0, asdict(root_span)) text = "".join( b.text for b in response.content if b.type == "text" ) return text

Pattern 3: Multi-Turn Conversation Tracing with Parent/Child Spans

In a real conversation, a user might send multiple messages. Each message triggers a new agent run, but they're all part of the same session. You need a session-level trace that contains multiple conversation-level traces.

session_tracer.py
import uuid import time import json from contextlib import contextmanager from typing import Generator, Any class SessionTracer: """Manages hierarchical trace context for multi-turn conversations.""" def __init__(self, session_id: str | None = None): self.session_id = session_id or uuid.uuid4().hex[:12] self.spans: list[dict] = [] self._span_stack: list[str] = [] # stack of active span IDs @contextmanager def span(self, name: str, kind: str = "generic", **attrs) -> Generator[dict, None, None]: """Context manager that creates and auto-closes a span.""" span_id = uuid.uuid4().hex[:16] parent_id = self._span_stack[-1] if self._span_stack else None span_data = { "span_id": span_id, "parent_id": parent_id, "session_id": self.session_id, "name": name, "kind": kind, "status": "ok", **attrs, } self._span_stack.append(span_id) t0 = time.perf_counter() try: yield span_data except Exception as e: span_data["status"] = "error" span_data["error"] = str(e) raise finally: span_data["latency_ms"] = round( (time.perf_counter() - t0) * 1000, 2 ) self.spans.append(span_data) self._span_stack.pop() def export_json(self) -> str: return json.dumps(self.spans, indent=2, default=str) # Usage example: # tracer = SessionTracer() # with tracer.span("conversation_turn", kind="agent") as root: # with tracer.span("llm_call", kind="llm", model="sonnet") as llm: # response = client.messages.create(...) # llm["input_tokens"] = response.usage.input_tokens # with tracer.span("tool:search", kind="tool") as tool: # result = search_documents(query="...") # tool["result_count"] = len(result)

Pattern 4: OpenTelemetry Instrumentation for Production

For production systems, you want your traces to flow into your standard observability stack. Here's how to instrument an agent with OpenTelemetry, using the emerging GenAI semantic conventions:

otel_agent.py
from opentelemetry import trace from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import ( OTLPSpanExporter, ) from opentelemetry.sdk.resources import Resource import anthropic # Setup: run once at application startup resource = Resource.create({ "service.name": "my-agent-service", "service.version": "1.0.0", "deployment.environment": "production", }) provider = TracerProvider(resource=resource) exporter = OTLPSpanExporter(endpoint="http://localhost:4317") provider.add_span_processor(BatchSpanProcessor(exporter)) trace.set_tracer_provider(provider) tracer = trace.get_tracer("agent-tracer") def traced_agent_call(user_message: str, model: str = "claude-sonnet-4-20250514"): client = anthropic.Anthropic() with tracer.start_as_current_span("agent_run") as root: root.set_attribute("gen_ai.system", "anthropic") root.set_attribute("user.message_preview", user_message[:200]) with tracer.start_as_current_span("llm_call") as llm_span: llm_span.set_attribute("gen_ai.request.model", model) llm_span.set_attribute("gen_ai.request.max_tokens", 4096) response = client.messages.create( model=model, max_tokens=4096, messages=[{"role": "user", "content": user_message}], ) llm_span.set_attribute( "gen_ai.usage.input_tokens", response.usage.input_tokens, ) llm_span.set_attribute( "gen_ai.usage.output_tokens", response.usage.output_tokens, ) llm_span.set_attribute( "gen_ai.response.finish_reason", response.stop_reason, ) root.set_attribute("agent.total_tokens", response.usage.input_tokens + response.usage.output_tokens) return response.content[0].text

Pattern 5: Braintrust Integration

Braintrust offers one of the cleanest APIs for combined tracing and evaluation. Their @traced decorator and logging API make it straightforward to capture traces and then score them:

braintrust_trace.py
import braintrust import anthropic # Initialize the logger — traces go to your Braintrust project logger = braintrust.init_logger(project="my-agent") client = anthropic.Anthropic() @braintrust.traced def search_documents(query: str) -> list[dict]: """Tool function — automatically traced by Braintrust.""" # Your search implementation here return [{"title": "Q2 Report", "snippet": "Revenue grew 15%..."}] @braintrust.traced def run_agent(user_message: str) -> str: """Main agent loop — the @traced decorator captures the full span tree.""" messages = [{"role": "user", "content": user_message}] tools = [{ "name": "search_documents", "description": "Search the document store", "input_schema": { "type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"], }, }] response = client.messages.create( model="claude-sonnet-4-20250514", max_tokens=4096, tools=tools, messages=messages, ) # Log usage metrics directly to Braintrust braintrust.current_span().log( metrics={ "input_tokens": response.usage.input_tokens, "output_tokens": response.usage.output_tokens, "model": "claude-sonnet-4-20250514", } ) # Handle tool use (simplified for example) if response.stop_reason == "tool_use": for block in response.content: if block.type == "tool_use": result = search_documents(**block.input) # The @traced decorator on search_documents # automatically creates a child span text_blocks = [b for b in response.content if b.type == "text"] return text_blocks[0].text if text_blocks else "" # After running, you can score the trace for evaluations # logger.log(output=result, scores={"quality": 0.9})

Key Instrumentation Principles

  • Instrument at the boundary: Trace every external call (LLM API, tool execution, database query). Internal logic between calls is usually not worth tracing individually.
  • Capture inputs and outputs: Without the actual prompt text and completion text, your traces are just timing data. Capture the full content, truncating only if necessary for storage.
  • Use context managers or decorators: Manual span start/stop is error-prone. Context managers guarantee spans are closed even when exceptions occur.
  • Don't forget error traces: Failed calls are the most valuable traces. Always capture the exception message and stack trace in the span metadata.
  • Make it opt-out, not opt-in: Tracing should be on by default. Use an environment variable like TRACING_ENABLED=false to disable, not TRACING_ENABLED=true to enable.
Implementation Tip

Start with Pattern 1 (the basic decorator) today. It takes five minutes to add and immediately gives you visibility into cost and latency. Evolve to Patterns 2-5 as your needs grow. Don't let perfect be the enemy of good: basic tracing now beats comprehensive tracing next quarter.

5 Debugging Agent Failures

When Your Agent Goes Wrong

Agents fail in ways that traditional software doesn't. They don't just throw exceptions; they make subtly wrong decisions, hallucinate tool parameters, get stuck in loops, and produce plausible-sounding but incorrect output. Traces are your X-ray machine. Here's how to use them.

Failure Mode 1: Wrong Tool Selection

The agent receives "What's the weather in Tokyo?" and calls search_documents instead of get_weather. In your trace, this shows up as a tool span with an unexpected tool_name. The fix might be in your tool descriptions, your system prompt, or your tool schema. Without the trace showing exactly which tools were available and which was chosen, you're guessing.

Failure Mode 2: Hallucinated Parameters

The agent calls the right tool but invents a parameter value. For example, search_files(directory="/home/user/documents/Q3") when no such directory exists. The tool returns an error or empty result, the agent retries with another fabricated path, and the loop begins. In your trace, you'd see the tool_args metadata revealing the hallucinated path.

Failure Mode 3: Infinite Loops

The most expensive failure mode. The agent calls a tool, gets an unsatisfying result, decides to try again with slightly different parameters, gets another unsatisfying result, and repeats. Your trace will show a growing chain of nearly identical spans. In a task dispatcher system with 30+ scheduled tasks, one looping task can consume your entire daily budget.

The Latency Waterfall

A visual waterfall shows each span as a horizontal bar, positioned by start time and sized by duration. This immediately reveals where time is spent:

Time (ms): 0 500 1000 1500 2000 2500 3000 3500 4000 agent_run |=================================================================| | | llm_call_1 |===========| | 0ms 820ms | | | tool:search |====| | 820ms 1140ms | | | llm_call_2 |===================| | 1140ms 2480ms | | | tool:read_doc |==| | 2480ms 2710ms | | | llm_call_3 |====================| 2710ms 4050ms BOTTLENECK: llm_call_2 (1340ms) - 33% of total time Contains 8,400 input tokens (context growing)

From this waterfall, you can immediately see that llm_call_2 is the bottleneck. It's slow because it's processing 8,400 input tokens (the original prompt plus the search results from the first tool call). This tells you: maybe you should truncate search results before passing them to the next LLM call.

Trace Replay: Re-running Failed Traces

One of the most powerful debugging techniques is trace replay: take a failed trace, extract the exact inputs at each step, and re-run them deterministically with temperature=0. This lets you reproduce the failure reliably.

The Debugging Checklist

Agent Failure Debugging Checklist
  • Check the trace waterfall for timeout or excessive duration on any span
  • Verify stop_reason on every LLM span: is it end_turn, tool_use, or max_tokens? A max_tokens stop often means truncated reasoning
  • Check for repeated tool calls with similar arguments (loop detection)
  • Verify tool arguments against the schema: are any values hallucinated?
  • Check tool results: did the tool return an error that the agent misinterpreted as data?
  • Compare input token counts across turns: are they growing? (context accumulation)
  • Check if prompt caching was expected but missed (unexpected cost spike)
  • Verify the model used matches the expected model (routing errors)
  • Check for error spans that were silently swallowed (tool errors the agent ignored)
  • Compare against a known-good trace for the same query type

Building a Trace Analyzer

Instead of manually inspecting every trace, build an analyzer that flags anomalies automatically. Here's a practical implementation:

trace_analyzer.py
from dataclasses import dataclass from typing import Any @dataclass class Anomaly: severity: str # "critical", "warning", "info" category: str # "cost", "loop", "error", "latency" message: str span_id: str | None = None class TraceAnalyzer: def __init__(self, config: dict | None = None): self.config = config or { "max_turns": 20, "max_cost_usd": 1.0, "max_latency_ms": 30000, "loop_threshold": 3, # same tool called N+ times "token_growth_ratio": 2.0, } def analyze(self, spans: list[dict]) -> list[Anomaly]: anomalies: list[Anomaly] = [] # 1. Check for tool call loops tool_calls = [s for s in spans if s.get("kind") == "tool"] tool_counts: dict[str, int] = {} for tc in tool_calls: name = tc.get("name", "unknown") tool_counts[name] = tool_counts.get(name, 0) + 1 for name, count in tool_counts.items(): if count >= self.config["loop_threshold"]: anomalies.append(Anomaly( severity="critical", category="loop", message=f"Tool '{name}' called {count}x (threshold: " f"{self.config['loop_threshold']})", )) # 2. Check cost total_cost = sum(s.get("cost_usd", 0) for s in spans) if total_cost > self.config["max_cost_usd"]: anomalies.append(Anomaly( severity="warning", category="cost", message=f"Trace cost ${total_cost:.4f} exceeds " f"${self.config['max_cost_usd']} threshold", )) # 3. Check for token growth (context accumulation) llm_spans = [s for s in spans if s.get("kind") == "llm"] if len(llm_spans) >= 2: first_input = llm_spans[0].get("input_tokens", 1) last_input = llm_spans[-1].get("input_tokens", 1) if first_input > 0: ratio = last_input / first_input if ratio > self.config["token_growth_ratio"]: anomalies.append(Anomaly( severity="warning", category="cost", message=f"Input tokens grew {ratio:.1f}x across " f"{len(llm_spans)} turns (context bloat)", )) # 4. Check for errors error_spans = [s for s in spans if s.get("status") == "error"] for es in error_spans: anomalies.append(Anomaly( severity="critical", category="error", message=f"Span '{es.get('name')}' failed: " f"{es.get('error', 'unknown')}", span_id=es.get("span_id"), )) return anomalies # Usage: # analyzer = TraceAnalyzer() # issues = analyzer.analyze(tracer.spans) # for issue in issues: # print(f"[{issue.severity}] {issue.category}: {issue.message}")

Comparing Good vs. Bad Traces

One of the most effective debugging techniques is side-by-side trace comparison. Take a trace where the agent succeeded and one where it failed for a similar query. Diff them systematically:

  • Tool sequence: Did the failing trace call tools in a different order? Did it call extra tools? Did it skip a tool the good trace used?
  • Tool arguments: Were the arguments to the same tool different? Was one set of arguments more specific?
  • Token counts: Did the failing trace have significantly more input tokens at any turn? This suggests context pollution.
  • Timing: Was any span in the failing trace unusually slow? Timeouts can cause cascading failures.
  • Error propagation: Did an early error go unhandled, causing downstream failures?

Cost Debugging: "Why Did This Query Cost $2.50?"

When a single query costs significantly more than expected, the trace tells you exactly where the money went. Sort spans by cost (or by input_tokens + output_tokens if you haven't computed cost) and look at the top offenders. Common causes:

  • Context accumulation: Each turn includes the full conversation history. By turn 15, you're sending 50,000 tokens per call. The trace shows input token counts growing linearly.
  • Large tool results: A tool returns a 10,000-word document, which gets included verbatim in the next prompt. The trace shows the tool result size in the span metadata.
  • Model escalation: Your router sent the query to Opus when Sonnet would have sufficed. The trace shows which model was used on each span.
  • Retry storms: An intermittent error caused three retries, each with the full context. The trace shows duplicate spans with growing latency.
Real-World Scenario: The $47 Day

In a Cowork-style system with 30+ scheduled tasks, a task called daily_report_generator hit an empty database. Instead of failing gracefully, it entered a retry loop: search, get empty result, ask Claude "I didn't find anything, let me try differently," search again with a slightly different query, repeat. Each retry accumulated the prior conversation history. After 34 retries, the final turn sent 89,000 input tokens to Opus. The trace analyzer above would have caught this after the 3rd retry with a "loop" anomaly. Without tracing, it took two days to find.

6 Production Monitoring

From Debugging to Dashboards

Debugging is reactive. You look at traces after something goes wrong. Monitoring is proactive: you watch aggregate metrics to catch problems before users report them. Production agent monitoring requires a specific set of metrics that traditional web service monitoring doesn't cover.

The Golden Signals for Agent Systems

Google's Site Reliability Engineering book defines four golden signals: latency, traffic, errors, and saturation. For LLM agents, we need to extend these with agent-specific signals:

SignalMetricTargetAlert Threshold
Latencyp50, p95, p99 response timep50 < 3s, p99 < 15sp99 > 30s for 5 min
Error RateFailed traces / total traces< 2%> 5% for 10 min
Cost per QueryAverage USD per trace$0.01 - $0.05> $0.20 average over 1 hour
Token EfficiencyOutput tokens / input tokens ratio0.1 - 0.5Ratio > 1.0 (unusual verbosity)
Tool Success RateSuccessful tool calls / total> 95%< 85% for 15 min
Turns per QueryAverage LLM calls per trace2 - 5> 10 average over 1 hour
Cache Hit RatePrompt cache hits / total calls> 40%< 20% (cache invalidated?)
Loop DetectionTraces with >3 same-tool calls0Any occurrence
Max Tokens Hit% of calls hitting max_tokens< 5%> 15% (responses being truncated)

Building a Monitoring Class

Here's a production-ready metrics collector that aggregates trace data into the golden signals above:

agent_monitor.py
import time import statistics from collections import defaultdict, deque from dataclasses import dataclass, field from typing import Callable @dataclass class AlertRule: name: str check: Callable[["AgentMonitor"], bool] message: str cooldown_sec: int = 300 last_fired: float = 0 class AgentMonitor: def __init__(self, window_size: int = 100): self.latencies: deque = deque(maxlen=window_size) self.costs: deque = deque(maxlen=window_size) self.errors: deque = deque(maxlen=window_size) self.turns_per_query: deque = deque(maxlen=window_size) self.tool_results: deque = deque(maxlen=window_size * 5) self.total_traces = 0 self.alert_handlers: list[Callable] = [] self.rules: list[AlertRule] = self._default_rules() def record_trace(self, spans: list[dict]): """Record metrics from a completed trace.""" self.total_traces += 1 root = next((s for s in spans if not s.get("parent_id")), None) if root: self.latencies.append(root.get("latency_ms", 0)) self.errors.append(1 if root.get("status") == "error" else 0) cost = sum(s.get("cost_usd", 0) for s in spans) self.costs.append(cost) llm_turns = sum(1 for s in spans if s.get("kind") == "llm") self.turns_per_query.append(llm_turns) for s in spans: if s.get("kind") == "tool": self.tool_results.append(s.get("status", "ok")) self._check_alerts() def get_dashboard(self) -> dict: """Return current metrics snapshot for a dashboard.""" lat = sorted(self.latencies) if self.latencies else [0] return { "total_traces": self.total_traces, "p50_latency_ms": lat[len(lat) // 2], "p99_latency_ms": lat[int(len(lat) * 0.99)], "error_rate": sum(self.errors) / max(len(self.errors), 1), "avg_cost_usd": statistics.mean(self.costs) if self.costs else 0, "avg_turns": statistics.mean(self.turns_per_query) if self.turns_per_query else 0, "tool_success_rate": ( sum(1 for r in self.tool_results if r == "ok") / max(len(self.tool_results), 1) ), } def _default_rules(self) -> list[AlertRule]: return [ AlertRule( name="high_error_rate", check=lambda m: ( sum(m.errors) / max(len(m.errors), 1) > 0.05 ), message="Error rate exceeds 5%", ), AlertRule( name="cost_spike", check=lambda m: ( statistics.mean(m.costs) > 0.20 if m.costs else False ), message="Average cost per query exceeds $0.20", ), ] def _check_alerts(self): now = time.time() for rule in self.rules: if (now - rule.last_fired > rule.cooldown_sec and rule.check(self)): rule.last_fired = now for handler in self.alert_handlers: handler(rule.name, rule.message)

SLOs for Agent Systems

Service Level Objectives for agents are tricky because "correct" is harder to define than for traditional APIs. A REST endpoint either returns the right data or it doesn't. An agent might return a response that's 80% correct, or correct but unhelpfully verbose, or technically correct but misses the user's intent.

Practical SLOs for agent systems:

  • Availability SLO: 99.5% of agent invocations complete without an unhandled exception. This is your floor. If the agent crashes, that's always a failure.
  • Latency SLO: 95% of queries complete in under 10 seconds, 99% in under 30 seconds. Adjust based on your use case. Interactive chat needs p99 < 10s. Batch processing can tolerate minutes.
  • Cost SLO: 99% of queries cost less than $0.50. The 1% allows for complex multi-tool queries. Anything exceeding $0.50 should be investigated.
  • Quality SLO (the hard one): This requires user feedback integration. Set a target like "90% of rated responses receive positive feedback." Track this with trace-correlated thumbs-up/thumbs-down signals.

User Feedback Integration

The most valuable signal in agent monitoring is user feedback, but only if it's correlated with traces. When a user clicks thumbs-down, you need to instantly link that feedback to the specific trace that generated the response. This lets you:

  • Pull up the exact trace and see what went wrong
  • Aggregate negative feedback by tool, model, or query type
  • Identify systematic issues (e.g., "all queries involving the calendar tool get negative feedback")
  • Build evaluation datasets from real failures

Implementation is straightforward: when generating a response, attach the trace_id to the response payload. When the user submits feedback, log it with the same trace_id. Your dashboard can then join these datasets.

Production Alerting Channels

Your monitoring system should alert through multiple channels based on severity:

  • Critical (P1): Error rate > 10%, complete agent downtime, or a single trace costing > $5. Route to PagerDuty / on-call phone.
  • Warning (P2): Error rate > 5%, cost spike > 2x baseline, latency p99 > 30s. Route to Slack #agent-alerts channel.
  • Info (P3): New failure patterns, cache hit rate drop, model routing anomalies. Route to daily digest email.
Production System Example: Learn Imagine v3

A production learning system running multiple agent types needs distinct monitoring per agent role. The research agent has a higher acceptable latency (30s) but lower cost tolerance ($0.10). The chat agent has strict latency requirements (5s p95) but can use cheaper models. The evaluation agent runs in batch mode and cares most about consistency. Each role gets its own SLO targets and alert thresholds, all fed from the same trace data.

7 Cost Optimization Through Tracing

Tracing as a Cost Reduction Tool

Tracing isn't just for debugging. It's your most powerful tool for cost optimization. Without per-query cost visibility, you're optimizing blind. With it, you can identify the 20% of queries consuming 80% of your budget and target them precisely.

Understanding Model Costs

First, internalize the cost landscape. Prices as of mid-2026 (always verify current pricing):

ModelInput (per 1M tokens)Output (per 1M tokens)Relative CostTypical Use
Claude Opus 4$15.00$75.001x (baseline)Complex reasoning, code generation
Claude Sonnet 4$3.00$15.000.2xGeneral purpose, tool use
Claude Haiku 4$0.80$4.000.05xClassification, simple tasks
GPT-4o$2.50$10.000.15xGeneral purpose alternative
GPT-4o-mini$0.15$0.600.01xHigh-volume, simple tasks

The cost difference between Haiku and Opus is 20x. That means routing even 30% of your traffic from Opus to Haiku could reduce your bill by 50%. But you can only identify which queries are Haiku-eligible if you have trace data showing query complexity.

Identifying Expensive Traces

The first step in cost optimization is understanding your cost distribution. Build a cost analyzer that ranks traces by expense and identifies patterns:

cost_analyzer.py
import statistics from collections import Counter, defaultdict from typing import Any MODEL_COSTS_PER_1K = { "claude-opus-4-20250514": (0.015, 0.075), "claude-sonnet-4-20250514": (0.003, 0.015), "claude-haiku-4-20250414": (0.0008, 0.004), } class CostAnalyzer: def __init__(self, traces: list[list[dict]]): """Each trace is a list of spans.""" self.traces = traces def cost_of_span(self, span: dict) -> float: model = span.get("model", "") inp = span.get("input_tokens", 0) out = span.get("output_tokens", 0) rates = MODEL_COSTS_PER_1K.get(model, (0, 0)) return (inp / 1000) * rates[0] + (out / 1000) * rates[1] def top_expensive_traces(self, n: int = 10) -> list[dict]: ranked = [] for trace in self.traces: total = sum(self.cost_of_span(s) for s in trace) trace_id = trace[0].get("trace_id", "?") if trace else "?" ranked.append({"trace_id": trace_id, "cost": total, "spans": len(trace)}) ranked.sort(key=lambda x: x["cost"], reverse=True) return ranked[:n] def cost_by_model(self) -> dict[str, float]: by_model: dict[str, float] = defaultdict(float) for trace in self.traces: for span in trace: model = span.get("model") if model: by_model[model] += self.cost_of_span(span) return dict(by_model) def model_routing_opportunities(self) -> list[dict]: """Find traces using expensive models that might work with cheaper ones. Heuristic: short input, low output, single tool call = probably Haiku-eligible. """ opportunities = [] for trace in self.traces: llm_spans = [s for s in trace if s.get("kind") == "llm"] for span in llm_spans: model = span.get("model", "") inp = span.get("input_tokens", 0) out = span.get("output_tokens", 0) if ("opus" in model or "sonnet" in model) \ and inp < 2000 and out < 500: savings = self.cost_of_span(span) - ( (inp / 1000) * 0.0008 + (out / 1000) * 0.004 ) opportunities.append({ "span_id": span.get("span_id"), "current_model": model, "suggested": "claude-haiku-4-20250414", "potential_savings": round(savings, 6), }) return opportunities

Prompt Caching Strategies

Anthropic's prompt caching can reduce input token costs by up to 90% for repeated prefixes. Tracing tells you whether caching is actually working:

  • Track cache_creation_input_tokens and cache_read_input_tokens in your spans. The Anthropic API returns these in the usage object. If cache_read_input_tokens is zero, your cache isn't being hit.
  • Common cache miss causes: System prompt changed between calls (even a single character invalidates the cache), conversation history order changed, tools list modified.
  • Optimization: Keep your system prompt and tools definition stable. Put the static parts first (they form the cache prefix). Put dynamic content (user message, conversation history) last.

In your traces, add a computed field: cache_hit_rate = cache_read_input_tokens / (cache_read_input_tokens + input_tokens). Dashboard this over time. A sudden drop means something invalidated your cache.

Token Waste Analysis

Common sources of token waste, identifiable through traces:

  • Over-long system prompts: Your system prompt is 3,000 tokens but most of it is rarely relevant. Trace analysis can show which parts of the system prompt are actually referenced in responses (by comparing system prompt content with response content). Trim the unused parts.
  • Unnecessary tool calls: The agent calls get_current_time on every query, even when time is irrelevant. Trace analytics showing tool call frequency per query type reveal this pattern.
  • Redundant context: The agent includes full search results when a summary would suffice. Traces show the tool result size vs. how much of it appears in the final response.
  • Conversation history bloat: In multi-turn conversations, each turn includes all previous turns. Traces show input token counts growing linearly per turn. Solutions: summarize older turns, use a sliding window, or use prompt caching aggressively.

A/B Testing Model Versions

Traces enable rigorous A/B testing of model versions. Route 10% of traffic to a new model, capture traces from both, and compare:

  • Quality: Correlate user feedback scores with model version
  • Cost: Compare average cost per trace between versions
  • Latency: Compare p50/p95/p99 latency
  • Tool use patterns: Does the new model use tools differently? More efficiently?
  • Error rates: Does the new model fail more often?

In a multi-model setup where you route between Opus, Sonnet, and Haiku based on query complexity, traces give you the ground truth for your routing decisions. If your router sends 40% of traffic to Opus but trace analysis shows 60% of those queries are simple enough for Sonnet, you've found a significant cost optimization.

Rule of Thumb: The 80/20 of Agent Costs

In most agent systems, 80% of cost comes from 20% of queries. These are typically queries that trigger multiple tool calls, use expensive models, or get stuck in retry loops. Trace your top 20% by cost and optimize those specifically. The remaining 80% of queries are usually cheap enough that optimizing them yields diminishing returns.

8 Building Your Observability Stack

Reference Architecture

Let's bring everything together. Here's a reference architecture for a production agent system with full observability, modeled after a Cowork-style setup with 30+ scheduled tasks, a command dispatcher, and multi-model routing:

PRODUCTION AGENT OBSERVABILITY STACK +------------------------------------------------------------------+ | USER REQUESTS | | (API, Scheduled Tasks, CLI, Webhooks) | +----------------------------+-------------------------------------+ | +----------v-----------+ | COMMAND DISPATCHER | | (Routes to agents) | | trace_id generated | +----------+-----------+ | +--------------------+--------------------+ | | | +-------v------+ +-------v------+ +-------v------+ | Research | | Chat | | Task | | Agent | | Agent | | Agent | | (Opus/Sonnet) | | (Sonnet) | | (Haiku) | +-------+------+ +-------+------+ +-------+------+ | | | +--------------------+--------------------+ | +----------v-----------+ | TRACE COLLECTOR | | (OTel Collector | | or SDK export) | +----------+-----------+ | +----------------+----------------+ | | +----------v-----------+ +-----------v----------+ | TRACE STORAGE | | METRICS AGGREGATOR | | (Grafana Tempo / | | (Prometheus / | | Phoenix / SaaS) | | custom) | +----------+------------+ +-----------+----------+ | | +----------v-----------+ +-----------v----------+ | TRACE VIEWER | | DASHBOARDS | | (Grafana / Phoenix | | (Grafana / | | UI / custom) | | custom) | +----------+------------+ +-----------+----------+ | | +----------------+----------------+ | +----------v-----------+ | ALERTING | | (Slack, PagerDuty, | | email) | +-----------------------+

Choosing the Right Tool for Your Scale

Here's a decision tree to guide your tool selection:

START: How many agent queries per day? | +-- < 100/day --> Helicone (zero setup) + manual log review | +-- 100-1,000/day --> Braintrust or LangSmith (managed SaaS) | Reason: you need dashboards but not infra | +-- 1,000-10,000/day --> Phoenix self-hosted + Grafana dashboards | Reason: SaaS costs add up, you need control | +-- > 10,000/day --> OpenTelemetry + Grafana Tempo + custom dashboards Reason: you need scale, customization, and integration with existing observability stack MODIFIER: Is data sensitivity high? (PII, PHI, financial) | +-- YES --> Self-hosted only (Phoenix, OTel, or Helicone self-hosted) | +-- NO --> SaaS is fine MODIFIER: Do you have an existing observability stack? | +-- YES (Datadog/Grafana/etc.) --> OTel + integrate with existing | +-- NO --> Standalone tool (Braintrust, LangSmith, Phoenix)

Self-Hosted vs. SaaS: The Tradeoffs

This decision deserves careful consideration because it affects cost, maintenance burden, and data governance:

  • SaaS (LangSmith, Braintrust, Helicone cloud): Zero infrastructure maintenance. Automatic updates. Shared community features. But: your prompt data leaves your infrastructure, which may violate compliance requirements. Costs scale linearly with volume. You're dependent on the vendor's uptime and roadmap.
  • Self-Hosted (Phoenix, OTel + Tempo, Helicone self-hosted): Full data control. Fixed infrastructure cost (doesn't scale with query volume once provisioned). You can customize everything. But: you own the ops burden. Upgrades, scaling, backups, and monitoring the monitoring system are all your responsibility.

My recommendation: start with SaaS to move fast, build a thin abstraction layer, and migrate to self-hosted when you hit one of these triggers: (a) your monthly SaaS bill exceeds the cost of a dedicated VM, (b) a compliance requirement demands data residency, or (c) you need customization the SaaS doesn't support.

OpenTelemetry as the Universal Standard

Regardless of which tool you choose, instrument with OpenTelemetry semantics. Here's why:

  • Portability: OTel traces can be exported to any compatible backend. Switch from Phoenix to Grafana Tempo to Datadog without changing your application code.
  • Ecosystem: OTel has auto-instrumentation libraries for HTTP clients, databases, and message queues. Your agent's non-LLM dependencies get traced for free.
  • Standards: The GenAI semantic conventions are being standardized. Early adoption means your traces will be compatible with future tooling.
  • Correlation: OTel lets you correlate LLM traces with infrastructure traces. An agent that's slow because the database is slow shows up as one connected trace, not two disconnected investigations.

Retention Policies

Traces are verbose. A single agent query might generate 5-20KB of trace data. At 10,000 queries/day, that's 50-200MB/day, or 1.5-6GB/month. Manageable, but it adds up.

Recommended retention strategy:

  • Hot storage (full detail, 7 days): Keep all spans, all metadata, all input/output text. This is your debugging window.
  • Warm storage (aggregated, 90 days): Keep span trees and metadata, but drop full prompt/completion text. Replace with summaries or hashes. This is your trend analysis window.
  • Cold storage (metrics only, 1 year): Keep only aggregated metrics: cost, latency, error counts, model usage. This is your capacity planning window.
  • Special retention: Keep error traces and traces with negative user feedback at full detail indefinitely. These are your most valuable debugging and training data.

Building Custom Dashboards

If you're using Grafana (which you should, because it's free and excellent), here are the panels for your agent observability dashboard:

  1. Top row: Four stat panels showing current error rate, p50 latency, average cost per query, and total traces in the last hour
  2. Second row: Time series of error rate and latency (p50, p95, p99) over the last 24 hours
  3. Third row: Bar chart of cost by model, pie chart of tool usage distribution
  4. Fourth row: Table of recent error traces (linked to trace viewer), table of most expensive traces
  5. Bottom row: Cache hit rate over time, turns-per-query distribution histogram

The Meta-Problem: Observability of Your Observability

There's a delicious irony in building observability for AI agents: the observability system itself needs monitoring. If your trace collector goes down, you lose visibility at the exact moment you need it most (because something is probably going wrong).

Practical measures:

  • Health checks: Monitor your trace collector's uptime independently of the traces it collects. A simple HTTP health check endpoint, monitored by an external service (UptimeRobot, Pingdom), is sufficient.
  • Queue depth monitoring: If you're using a batch span processor (and you should be, for performance), monitor the queue depth. A growing queue means you're producing traces faster than you can export them.
  • Sampling fallback: If trace export fails, don't lose the data silently. Fall back to writing spans to a local file that can be replayed later. Better to have a big log file than no traces.
  • Separate critical path: Your trace export should never block your agent's response path. Use async export with a bounded queue. If the queue is full, drop spans (with a counter metric), don't slow down the agent.

Putting It All Together: Your First Week

If you're starting from zero, here's a practical one-week plan to go from no observability to a production-grade setup:

  1. Day 1: Add the basic trace decorator (Module 4, Pattern 1) to your agent's LLM calls. Deploy. You now have latency and cost visibility.
  2. Day 2: Add tool call tracing (Pattern 2). Deploy. You now have full span trees.
  3. Day 3: Set up a trace backend. If SaaS: sign up for Braintrust or LangSmith, add their SDK. If self-hosted: pip install arize-phoenix and phoenix serve. Export your spans.
  4. Day 4: Build the trace analyzer (Module 5). Add it as a post-processing step. Log anomalies to your team's Slack channel.
  5. Day 5: Set up the monitoring class (Module 6). Create a Grafana dashboard (or use your trace tool's built-in dashboard) with the golden signals.
  6. Day 6: Run the cost analyzer (Module 7) on your first week of data. Identify the top 10 most expensive traces. Fix the obvious wins (loops, model routing).
  7. Day 7: Write documentation for your team. Add alerting rules. Celebrate: you've gone from black box to glass box in one week.
Final Thought

The best observability system is the one you actually use. Don't over-engineer your first version. Start with the basic decorator, iterate based on what you learn from the data, and invest in more sophisticated tooling only when the data tells you where to look. Your agents are already running. Start watching them today.

Need this for a date?

Turn this course into a ramp-up pack sized to your minutes per day, or build an interview or certification pack for the day you need it.