Agent Tracing & Observability
From Black Box to Glass Box: Making Your AI Agents Observable
You're running agents in production. Maybe it's a Cowork-style setup with 30+ scheduled tasks, a command dispatcher, and multi-model routing. Maybe it's a customer-facing assistant that makes tool calls you can't see. Either way, your agents are black boxes, and that's terrifying. This course takes you from zero observability to a production-grade tracing stack, with real Python code you can deploy today. No hand-waving. No vendor marketing. Just engineering.
Spans, Traces, and Events
Before you instrument anything, you need to understand the data model. Tracing has precise terminology, and using it correctly will save you hours of confusion when you're reading documentation for any tracing tool.
A trace represents a single end-to-end operation. In agent terms, one trace equals one user query from the moment it arrives to the moment the final response is returned. A trace has a unique trace_id and contains one or more spans.
A span represents a single unit of work within a trace. Spans have a start time, an end time, a name, and optional metadata. Critically, spans have parent-child relationships. The root span is the top-level operation. Child spans represent sub-operations. This gives you a tree structure.
An event (sometimes called a "log" in OpenTelemetry terminology) is a timestamped annotation attached to a span. Events don't have duration; they mark a point in time. "Tool call started," "Cache hit," "Retry initiated" are events.
Here's how an agent query decomposes into spans:
Notice the structure. The root span encompasses the entire operation. Each LLM call and tool execution is a child span. Events are annotations within spans. You can see the data flow: the first LLM call decided to search, the tool returned results, the second LLM call decided to read a specific document, the tool returned content, and the third LLM call generated the summary.
The Span Data Model in Python
Let's formalize this. Here's a Python dataclass that models a span suitable for LLM agent tracing:
from dataclasses import dataclass, field
from typing import Optional, Any
from datetime import datetime
from enum import Enum
import uuid
class SpanKind(Enum):
AGENT_RUN = "agent_run"
LLM_CALL = "llm_call"
TOOL_EXEC = "tool_execution"
RETRIEVAL = "retrieval"
GUARDRAIL = "guardrail_check"
@dataclass
class SpanEvent:
name: str
timestamp: datetime = field(default_factory=datetime.utcnow)
attributes: dict[str, Any] = field(default_factory=dict)
@dataclass
class Span:
# Identity
span_id: str = field(default_factory=lambda: uuid.uuid4().hex[:16])
trace_id: str = ""
parent_span_id: Optional[str] = None
name: str = ""
kind: SpanKind = SpanKind.AGENT_RUN
# Timing
start_time: datetime = field(default_factory=datetime.utcnow)
end_time: Optional[datetime] = None
# LLM-specific metadata
model: Optional[str] = None
input_tokens: int = 0
output_tokens: int = 0
cost_usd: float = 0.0
temperature: Optional[float] = None
# Input/Output capture
input_data: Optional[Any] = None # prompt or tool args
output_data: Optional[Any] = None # completion or tool result
# Tool-specific
tool_name: Optional[str] = None
tool_args: Optional[dict] = None
tool_result: Optional[Any] = None
# Status
status: str = "ok" # ok, error, timeout
error_message: Optional[str] = None
# Events and children
events: list[SpanEvent] = field(default_factory=list)
children: list["Span"] = field(default_factory=list)
def duration_ms(self) -> Optional[float]:
if self.end_time and self.start_time:
return (self.end_time - self.start_time).total_seconds() * 1000
return None
def total_cost(self) -> float:
return self.cost_usd + sum(c.total_cost() for c in self.children)
Standard Metadata Fields
Every tracing tool uses slightly different field names, but the following metadata should be captured on every LLM span. This table serves as your instrumentation checklist:
| Field | Type | Description | Why It Matters |
|---|---|---|---|
trace_id | string | Unique ID for the entire operation | Links all spans in one query together |
span_id | string | Unique ID for this span | Identifies this specific unit of work |
parent_span_id | string? | ID of the parent span | Builds the trace tree |
model | string | Model used (claude-sonnet-4-20250514, gpt-4o, etc.) | Cost calculation, performance comparison |
input_tokens | int | Tokens in the prompt | Cost tracking, context window monitoring |
output_tokens | int | Tokens in the completion | Cost tracking, verbosity analysis |
latency_ms | float | Wall-clock time for this span | Performance optimization, SLO monitoring |
temperature | float? | Sampling temperature used | Debugging non-deterministic behavior |
tool_name | string? | Name of tool called | Tool usage analytics, error attribution |
tool_args | dict? | Arguments passed to the tool | Debugging hallucinated parameters |
tool_result | any? | Result returned by the tool | Debugging incorrect tool behavior |
finish_reason | string | Why the LLM stopped (end_turn, tool_use, max_tokens) | Detecting truncated responses |
cache_hit | bool? | Whether prompt caching was used | Cost optimization verification |
error | string? | Error message if span failed | Failure debugging |
user_id | string? | User who initiated the trace | Per-user cost and error attribution |
session_id | string? | Conversation session ID | Multi-turn conversation tracking |
Trace ID Propagation in Multi-Agent Systems
In simple setups, one agent handles one query. In production systems, you often have orchestrator agents that delegate to specialist agents. A command dispatcher might route a request to a "research agent," which calls a "summarization agent," which calls a "citation verification agent." Each agent might run in a different process or even a different service.
Trace ID propagation ensures that all spans from all agents in a single user query share the same trace_id. Without this, you see five disconnected traces instead of one coherent tree.
The standard approach, borrowed from OpenTelemetry, is to pass trace context via headers or function arguments:
import contextvars
# Thread-local trace context using contextvars
_current_trace_id: contextvars.ContextVar[str] = contextvars.ContextVar("trace_id")
_current_span_id: contextvars.ContextVar[str] = contextvars.ContextVar("span_id")
def get_trace_context() -> dict:
"""Get current trace context for propagation."""
return {
"trace_id": _current_trace_id.get(None),
"parent_span_id": _current_span_id.get(None),
}
def set_trace_context(trace_id: str, span_id: str):
"""Inject trace context (e.g., when receiving from another service)."""
_current_trace_id.set(trace_id)
_current_span_id.set(span_id)
How OpenTelemetry Maps to LLM Agent Concepts
OpenTelemetry (OTel) is the industry standard for distributed tracing. Its data model maps cleanly to LLM agents, but requires some semantic conventions that aren't yet standardized (there is an active working group for "GenAI semantic conventions"):
- OTel Trace = Agent conversation or task execution
- OTel Span = LLM call, tool execution, or agent reasoning step
- OTel SpanKind.CLIENT = Outgoing LLM API call
- OTel SpanKind.INTERNAL = Agent's decision-making logic
- OTel Attributes = Model name, token counts, cost, tool metadata
- OTel Events = Tool selection, cache hits, guardrail checks
- OTel Links = References between related traces (e.g., a follow-up query linked to the original)
The emerging convention uses attribute names like gen_ai.system (e.g., "anthropic"), gen_ai.request.model, gen_ai.usage.input_tokens, and gen_ai.usage.output_tokens. If you adopt these now, your traces will be compatible with any OTel-based backend.
When in doubt, capture more metadata rather than less. Storage is cheap; missing data during an incident is expensive. Capture the full prompt and completion text (with PII redaction if needed), all tool arguments, all tool results. You can always add filtering and retention policies later. You can't retroactively add data you didn't collect.
The Observability Tools Ecosystem in 2026
The LLM observability space has matured rapidly. In 2023, you had LangSmith and not much else. In 2026, there are credible options across the spectrum from open-source self-hosted to fully managed SaaS. Choosing the right tool depends on your team size, budget, compliance requirements, and existing infrastructure.
I'm going to give you my honest, opinionated assessment of each major tool. I've used most of these in production or evaluated them seriously. Where I haven't, I'll say so.
LangSmith (by LangChain)
LangSmith is the most widely adopted LLM tracing tool, largely because of LangChain's market dominance in agent frameworks. If you're already using LangChain or LangGraph, LangSmith integrates with zero additional code. You literally import it and traces appear.
Strengths: Excellent UI for trace visualization, built-in evaluation framework, dataset management for testing, hub for sharing prompts. The trace tree view is best-in-class. Playground for iterating on prompts with trace data.
Weaknesses: Tightly coupled to the LangChain ecosystem. If you're using the Anthropic or OpenAI SDKs directly, integration requires more work. Pricing can surprise you at scale. The platform has had reliability issues under heavy load. Not open source (LangSmith is SaaS, though LangChain the framework is open source).
My take: If you're a LangChain shop, just use it. If you're not, look elsewhere.
Braintrust
Braintrust started as an evaluation platform and grew into tracing. This heritage shows in the best ways: their eval framework is exceptional, and traces integrate directly with eval scores. You can trace a production query, add it to a dataset, run evals against it, and compare model versions, all in one platform.
Strengths: Best-in-class evaluation integration. Clean SDK that works with any LLM provider. Excellent logging API that's framework-agnostic. Good proxy mode for zero-code tracing. Strong cost analytics.
Weaknesses: Smaller community than LangSmith. Some advanced features require the paid tier. The UI, while functional, isn't as polished as LangSmith's trace viewer.
My take: If evaluation is your primary concern alongside tracing, Braintrust is the best choice. The combined eval+trace workflow is unmatched.
Arize Phoenix
Phoenix is the open-source option from Arize AI. It's a standalone tool you run locally or self-host. It uses OpenTelemetry under the hood, which means your instrumentation is portable: if you outgrow Phoenix, your traces work with any OTel-compatible backend.
Strengths: Fully open source (Apache 2.0). Self-hosted, so your data never leaves your infrastructure. OTel-native, so you're not locked in. Good for compliance-sensitive environments. Free. Active community.
Weaknesses: You own the infrastructure. No managed service (though Arize's commercial platform offers one). The UI is functional but spartan compared to commercial tools. Less polished eval integration.
My take: If you need self-hosted and open source, Phoenix is the clear winner. It's also a great starting point if you're not sure what you need yet, since it's free and OTel-based.
Helicone
Helicone takes a unique approach: it's a proxy. You change your API base URL from api.anthropic.com to anthropic.helicone.ai, and every request is logged automatically. Zero code changes to your application.
Strengths: Literally zero-code setup for basic tracing. Works with any SDK or framework. Excellent cost dashboard. Good for getting started quickly. Rate limiting and caching features built in.
Weaknesses: Being a proxy adds latency (typically 10-50ms). You're routing all your LLM traffic through a third party. Limited ability to trace non-LLM operations (tool calls, custom logic). The proxy model makes it hard to capture tool execution details.
My take: Helicone is perfect for teams that want immediate visibility with zero engineering effort. It's not sufficient for deep agent debugging, but it's a great complement to a more comprehensive solution.
Weights & Biases Weave
W&B Weave brings the maturity of W&B's ML experiment tracking to LLM observability. If your team already uses W&B for model training, Weave provides a unified platform for both traditional ML and LLM ops.
Strengths: Integration with W&B's broader ML platform. Good versioning for prompts and models. Strong visualization. Enterprise features (SSO, audit logs, etc.).
Weaknesses: The LLM-specific features feel bolted onto a platform designed for traditional ML. Pricing can be opaque. The learning curve is steeper if you're not already a W&B user.
My take: Use it if you're already in the W&B ecosystem. Don't adopt W&B just for LLM tracing.
OpenTelemetry (Direct)
You can skip all vendor tools and instrument your agents with raw OpenTelemetry. Send traces to any OTel-compatible backend: Jaeger, Grafana Tempo, Honeycomb, Datadog, or your own collector. This gives you maximum flexibility and zero vendor lock-in.
Strengths: Industry standard. No vendor lock-in. Works with your existing observability stack. You control everything.
Weaknesses: You build everything yourself. No pre-built LLM trace viewer. No eval integration. No cost analytics dashboard. Significant engineering investment.
My take: This is the right choice for platform teams at scale (50+ engineers) who want to integrate LLM tracing into their existing observability stack. It's overkill for small teams.
Anthropic's Usage API
Anthropic provides usage tracking through the API response's usage field and the admin API. It's not tracing in the traditional sense, but it gives you token counts, costs, and rate limit information per request.
Strengths: Built in. No additional setup. Accurate token counts straight from the source.
Weaknesses: Not a tracing tool. No trace visualization, no span trees, no eval integration. Just per-request metrics.
My take: Always capture the usage data from API responses. Layer a real tracing tool on top.
Comprehensive Comparison
| Tool | Pricing | Setup | Key Features | OSS | Self-Host | Best For |
|---|---|---|---|---|---|---|
| LangSmith | Free tier, then $39+/seat/mo | Low (LangChain), Medium (other) | Trace UI, evals, datasets, playground | No | Enterprise only | LangChain teams |
| Braintrust | Free tier, usage-based after | Low | Evals + tracing, proxy mode, scoring | No | No | Eval-focused teams |
| Arize Phoenix | Free (OSS) | Medium | OTel-native, self-hosted, trace viewer | Yes | Yes | Privacy-sensitive, budget-conscious |
| Helicone | Free tier, then usage-based | Very Low | Proxy-based, cost dashboard, caching | Yes | Yes | Quick wins, cost monitoring |
| W&B Weave | Free tier, then $50+/seat/mo | Medium | ML platform integration, versioning | No | Enterprise only | Existing W&B users |
| OpenTelemetry | Free (your infra costs) | High | Universal standard, any backend | Yes | Yes | Platform teams at scale |
| Anthropic Usage | Free (included) | Zero | Token counts, cost per request | N/A | N/A | Baseline metrics |
Build vs. Buy: The Decision Framework
The build vs. buy decision for LLM tracing depends on three factors:
- Team size: Under 10 engineers? Buy. The engineering cost of building and maintaining a tracing system exceeds the subscription cost of any SaaS tool. 50+ engineers with a platform team? Build on OTel, potentially with an open-source frontend like Phoenix.
- Data sensitivity: If your prompts contain PII, PHI, or proprietary data, self-hosted is often a compliance requirement. This narrows your options to Phoenix, Helicone (self-hosted), or raw OTel.
- Existing infrastructure: If you already have Datadog or Grafana, adding OTel-based LLM tracing to your existing dashboards is often simpler than adopting a new SaaS tool. Your ops team already knows how to manage alerts, dashboards, and retention in the existing system.
Opinionated Recommendations by Team Size
Solo developer or small startup (1-5 people): Start with Helicone for instant cost visibility. Add Braintrust when you need evals. Total cost: $0-50/month.
Growth-stage team (5-20 people): Use Braintrust or LangSmith as your primary platform. Capture everything. The investment in trace infrastructure will pay back 10x when you're debugging production issues at 2 AM. Total cost: $200-500/month.
Enterprise / platform team (20+ people): Build on OpenTelemetry. Use Phoenix or a commercial OTel backend (Honeycomb, Grafana Cloud). Integrate with your existing observability stack. Build custom dashboards. This is a 2-4 week engineering project but gives you maximum control and zero vendor lock-in.
Whatever tool you choose, instrument your code with a thin abstraction layer. Don't scatter langsmith.trace() calls throughout your codebase. Write a Tracer interface, implement it for your chosen backend, and swap implementations later without touching application code. You will switch tools at least once. Make it easy.
From Theory to Code
This is the hands-on module. We're going to build real instrumentation for real agent patterns. Every code example is designed to be copied into a production codebase with minimal modification. I'll cover five patterns, from simple to complex.
Pattern 1: A Basic Trace Decorator
The simplest useful instrumentation is a decorator that wraps any LLM call, captures timing, token usage, and cost, and stores the span. This works with any LLM provider.
import time
import uuid
import functools
import json
from datetime import datetime
from typing import Callable, Any
# In-memory span storage (replace with your backend)
_spans: list[dict] = []
# Cost per 1K tokens (update with current pricing)
MODEL_COSTS = {
"claude-sonnet-4-20250514": {"input": 0.003, "output": 0.015},
"claude-haiku-4-20250414": {"input": 0.0008, "output": 0.004},
"claude-opus-4-20250514": {"input": 0.015, "output": 0.075},
"gpt-4o": {"input": 0.0025, "output": 0.01},
}
def calc_cost(model: str, input_tok: int, output_tok: int) -> float:
costs = MODEL_COSTS.get(model, {"input": 0, "output": 0})
return (input_tok / 1000) * costs["input"] + (output_tok / 1000) * costs["output"]
def trace_llm(model: str, name: str = "llm_call"):
"""Decorator to trace any function that returns an LLM response."""
def decorator(func: Callable) -> Callable:
@functools.wraps(func)
def wrapper(*args, **kwargs) -> Any:
span = {
"span_id": uuid.uuid4().hex[:16],
"name": name,
"model": model,
"start_time": datetime.utcnow().isoformat(),
"input_preview": str(args)[:500],
}
start = time.perf_counter()
try:
result = func(*args, **kwargs)
elapsed = (time.perf_counter() - start) * 1000
# Extract usage from Anthropic/OpenAI response
usage = getattr(result, "usage", None)
if usage:
span["input_tokens"] = usage.input_tokens
span["output_tokens"] = usage.output_tokens
span["cost_usd"] = calc_cost(
model, usage.input_tokens, usage.output_tokens
)
span["latency_ms"] = round(elapsed, 2)
span["status"] = "ok"
_spans.append(span)
return result
except Exception as e:
span["status"] = "error"
span["error"] = str(e)
span["latency_ms"] = round(
(time.perf_counter() - start) * 1000, 2
)
_spans.append(span)
raise
return wrapper
return decorator
Pattern 2: Tracing Claude Messages API with Tool Use
This is the pattern you'll use most often: tracing a Claude conversation that includes tool use. The key insight is that you need to trace both the LLM calls and the tool executions, and connect them with parent-child relationships.
import anthropic
import time
import uuid
from dataclasses import dataclass, field, asdict
from typing import Any
@dataclass
class TracedSpan:
span_id: str = field(default_factory=lambda: uuid.uuid4().hex[:16])
parent_id: str | None = None
name: str = ""
kind: str = "generic"
start: float = field(default_factory=time.perf_counter)
metadata: dict = field(default_factory=dict)
class TracedAgent:
def __init__(self, tools: list[dict], tool_handlers: dict):
self.client = anthropic.Anthropic()
self.tools = tools
self.tool_handlers = tool_handlers
self.spans: list[dict] = []
def run(self, user_msg: str, model: str = "claude-sonnet-4-20250514") -> str:
trace_id = uuid.uuid4().hex[:16]
root_span = TracedSpan(name="agent_run", kind="root")
messages = [{"role": "user", "content": user_msg}]
turn = 0
while True:
turn += 1
# --- Trace the LLM call ---
llm_span = TracedSpan(
name=f"llm_call_{turn}",
kind="llm",
parent_id=root_span.span_id,
)
t0 = time.perf_counter()
response = self.client.messages.create(
model=model,
max_tokens=4096,
tools=self.tools,
messages=messages,
)
llm_span.metadata = {
"model": model,
"input_tokens": response.usage.input_tokens,
"output_tokens": response.usage.output_tokens,
"stop_reason": response.stop_reason,
"latency_ms": round((time.perf_counter() - t0) * 1000),
"trace_id": trace_id,
}
self.spans.append(asdict(llm_span))
# --- Handle tool use ---
if response.stop_reason == "tool_use":
tool_blocks = [b for b in response.content
if b.type == "tool_use"]
tool_results = []
for block in tool_blocks:
tool_span = TracedSpan(
name=f"tool:{block.name}",
kind="tool",
parent_id=llm_span.span_id,
)
t1 = time.perf_counter()
try:
result = self.tool_handlers[block.name](**block.input)
tool_span.metadata = {
"tool": block.name,
"args": block.input,
"result_preview": str(result)[:500],
"status": "ok",
"latency_ms": round(
(time.perf_counter() - t1) * 1000
),
}
except Exception as e:
result = f"Error: {e}"
tool_span.metadata = {
"tool": block.name,
"args": block.input,
"error": str(e),
"status": "error",
}
self.spans.append(asdict(tool_span))
tool_results.append({
"type": "tool_result",
"tool_use_id": block.id,
"content": str(result),
})
messages.append({"role": "assistant", "content": response.content})
messages.append({"role": "user", "content": tool_results})
else:
# Final text response
root_span.metadata = {
"trace_id": trace_id,
"total_turns": turn,
"total_spans": len(self.spans),
"latency_ms": round(
(time.perf_counter() - root_span.start) * 1000
),
}
self.spans.insert(0, asdict(root_span))
text = "".join(
b.text for b in response.content if b.type == "text"
)
return text
Pattern 3: Multi-Turn Conversation Tracing with Parent/Child Spans
In a real conversation, a user might send multiple messages. Each message triggers a new agent run, but they're all part of the same session. You need a session-level trace that contains multiple conversation-level traces.
import uuid
import time
import json
from contextlib import contextmanager
from typing import Generator, Any
class SessionTracer:
"""Manages hierarchical trace context for multi-turn conversations."""
def __init__(self, session_id: str | None = None):
self.session_id = session_id or uuid.uuid4().hex[:12]
self.spans: list[dict] = []
self._span_stack: list[str] = [] # stack of active span IDs
@contextmanager
def span(self, name: str, kind: str = "generic",
**attrs) -> Generator[dict, None, None]:
"""Context manager that creates and auto-closes a span."""
span_id = uuid.uuid4().hex[:16]
parent_id = self._span_stack[-1] if self._span_stack else None
span_data = {
"span_id": span_id,
"parent_id": parent_id,
"session_id": self.session_id,
"name": name,
"kind": kind,
"status": "ok",
**attrs,
}
self._span_stack.append(span_id)
t0 = time.perf_counter()
try:
yield span_data
except Exception as e:
span_data["status"] = "error"
span_data["error"] = str(e)
raise
finally:
span_data["latency_ms"] = round(
(time.perf_counter() - t0) * 1000, 2
)
self.spans.append(span_data)
self._span_stack.pop()
def export_json(self) -> str:
return json.dumps(self.spans, indent=2, default=str)
# Usage example:
# tracer = SessionTracer()
# with tracer.span("conversation_turn", kind="agent") as root:
# with tracer.span("llm_call", kind="llm", model="sonnet") as llm:
# response = client.messages.create(...)
# llm["input_tokens"] = response.usage.input_tokens
# with tracer.span("tool:search", kind="tool") as tool:
# result = search_documents(query="...")
# tool["result_count"] = len(result)
Pattern 4: OpenTelemetry Instrumentation for Production
For production systems, you want your traces to flow into your standard observability stack. Here's how to instrument an agent with OpenTelemetry, using the emerging GenAI semantic conventions:
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (
OTLPSpanExporter,
)
from opentelemetry.sdk.resources import Resource
import anthropic
# Setup: run once at application startup
resource = Resource.create({
"service.name": "my-agent-service",
"service.version": "1.0.0",
"deployment.environment": "production",
})
provider = TracerProvider(resource=resource)
exporter = OTLPSpanExporter(endpoint="http://localhost:4317")
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("agent-tracer")
def traced_agent_call(user_message: str, model: str = "claude-sonnet-4-20250514"):
client = anthropic.Anthropic()
with tracer.start_as_current_span("agent_run") as root:
root.set_attribute("gen_ai.system", "anthropic")
root.set_attribute("user.message_preview", user_message[:200])
with tracer.start_as_current_span("llm_call") as llm_span:
llm_span.set_attribute("gen_ai.request.model", model)
llm_span.set_attribute("gen_ai.request.max_tokens", 4096)
response = client.messages.create(
model=model,
max_tokens=4096,
messages=[{"role": "user", "content": user_message}],
)
llm_span.set_attribute(
"gen_ai.usage.input_tokens",
response.usage.input_tokens,
)
llm_span.set_attribute(
"gen_ai.usage.output_tokens",
response.usage.output_tokens,
)
llm_span.set_attribute(
"gen_ai.response.finish_reason",
response.stop_reason,
)
root.set_attribute("agent.total_tokens",
response.usage.input_tokens + response.usage.output_tokens)
return response.content[0].text
Pattern 5: Braintrust Integration
Braintrust offers one of the cleanest APIs for combined tracing and evaluation. Their @traced decorator and logging API make it straightforward to capture traces and then score them:
import braintrust
import anthropic
# Initialize the logger — traces go to your Braintrust project
logger = braintrust.init_logger(project="my-agent")
client = anthropic.Anthropic()
@braintrust.traced
def search_documents(query: str) -> list[dict]:
"""Tool function — automatically traced by Braintrust."""
# Your search implementation here
return [{"title": "Q2 Report", "snippet": "Revenue grew 15%..."}]
@braintrust.traced
def run_agent(user_message: str) -> str:
"""Main agent loop — the @traced decorator captures the full span tree."""
messages = [{"role": "user", "content": user_message}]
tools = [{
"name": "search_documents",
"description": "Search the document store",
"input_schema": {
"type": "object",
"properties": {"query": {"type": "string"}},
"required": ["query"],
},
}]
response = client.messages.create(
model="claude-sonnet-4-20250514",
max_tokens=4096,
tools=tools,
messages=messages,
)
# Log usage metrics directly to Braintrust
braintrust.current_span().log(
metrics={
"input_tokens": response.usage.input_tokens,
"output_tokens": response.usage.output_tokens,
"model": "claude-sonnet-4-20250514",
}
)
# Handle tool use (simplified for example)
if response.stop_reason == "tool_use":
for block in response.content:
if block.type == "tool_use":
result = search_documents(**block.input)
# The @traced decorator on search_documents
# automatically creates a child span
text_blocks = [b for b in response.content if b.type == "text"]
return text_blocks[0].text if text_blocks else ""
# After running, you can score the trace for evaluations
# logger.log(output=result, scores={"quality": 0.9})
Key Instrumentation Principles
- Instrument at the boundary: Trace every external call (LLM API, tool execution, database query). Internal logic between calls is usually not worth tracing individually.
- Capture inputs and outputs: Without the actual prompt text and completion text, your traces are just timing data. Capture the full content, truncating only if necessary for storage.
- Use context managers or decorators: Manual span start/stop is error-prone. Context managers guarantee spans are closed even when exceptions occur.
- Don't forget error traces: Failed calls are the most valuable traces. Always capture the exception message and stack trace in the span metadata.
- Make it opt-out, not opt-in: Tracing should be on by default. Use an environment variable like
TRACING_ENABLED=falseto disable, notTRACING_ENABLED=trueto enable.
Start with Pattern 1 (the basic decorator) today. It takes five minutes to add and immediately gives you visibility into cost and latency. Evolve to Patterns 2-5 as your needs grow. Don't let perfect be the enemy of good: basic tracing now beats comprehensive tracing next quarter.
When Your Agent Goes Wrong
Agents fail in ways that traditional software doesn't. They don't just throw exceptions; they make subtly wrong decisions, hallucinate tool parameters, get stuck in loops, and produce plausible-sounding but incorrect output. Traces are your X-ray machine. Here's how to use them.
Failure Mode 1: Wrong Tool Selection
The agent receives "What's the weather in Tokyo?" and calls search_documents instead of get_weather. In your trace, this shows up as a tool span with an unexpected tool_name. The fix might be in your tool descriptions, your system prompt, or your tool schema. Without the trace showing exactly which tools were available and which was chosen, you're guessing.
Failure Mode 2: Hallucinated Parameters
The agent calls the right tool but invents a parameter value. For example, search_files(directory="/home/user/documents/Q3") when no such directory exists. The tool returns an error or empty result, the agent retries with another fabricated path, and the loop begins. In your trace, you'd see the tool_args metadata revealing the hallucinated path.
Failure Mode 3: Infinite Loops
The most expensive failure mode. The agent calls a tool, gets an unsatisfying result, decides to try again with slightly different parameters, gets another unsatisfying result, and repeats. Your trace will show a growing chain of nearly identical spans. In a task dispatcher system with 30+ scheduled tasks, one looping task can consume your entire daily budget.
The Latency Waterfall
A visual waterfall shows each span as a horizontal bar, positioned by start time and sized by duration. This immediately reveals where time is spent:
From this waterfall, you can immediately see that llm_call_2 is the bottleneck. It's slow because it's processing 8,400 input tokens (the original prompt plus the search results from the first tool call). This tells you: maybe you should truncate search results before passing them to the next LLM call.
Trace Replay: Re-running Failed Traces
One of the most powerful debugging techniques is trace replay: take a failed trace, extract the exact inputs at each step, and re-run them deterministically with temperature=0. This lets you reproduce the failure reliably.
The Debugging Checklist
- Check the trace waterfall for timeout or excessive duration on any span
- Verify
stop_reasonon every LLM span: is itend_turn,tool_use, ormax_tokens? Amax_tokensstop often means truncated reasoning - Check for repeated tool calls with similar arguments (loop detection)
- Verify tool arguments against the schema: are any values hallucinated?
- Check tool results: did the tool return an error that the agent misinterpreted as data?
- Compare input token counts across turns: are they growing? (context accumulation)
- Check if prompt caching was expected but missed (unexpected cost spike)
- Verify the model used matches the expected model (routing errors)
- Check for error spans that were silently swallowed (tool errors the agent ignored)
- Compare against a known-good trace for the same query type
Building a Trace Analyzer
Instead of manually inspecting every trace, build an analyzer that flags anomalies automatically. Here's a practical implementation:
from dataclasses import dataclass
from typing import Any
@dataclass
class Anomaly:
severity: str # "critical", "warning", "info"
category: str # "cost", "loop", "error", "latency"
message: str
span_id: str | None = None
class TraceAnalyzer:
def __init__(self, config: dict | None = None):
self.config = config or {
"max_turns": 20,
"max_cost_usd": 1.0,
"max_latency_ms": 30000,
"loop_threshold": 3, # same tool called N+ times
"token_growth_ratio": 2.0,
}
def analyze(self, spans: list[dict]) -> list[Anomaly]:
anomalies: list[Anomaly] = []
# 1. Check for tool call loops
tool_calls = [s for s in spans if s.get("kind") == "tool"]
tool_counts: dict[str, int] = {}
for tc in tool_calls:
name = tc.get("name", "unknown")
tool_counts[name] = tool_counts.get(name, 0) + 1
for name, count in tool_counts.items():
if count >= self.config["loop_threshold"]:
anomalies.append(Anomaly(
severity="critical",
category="loop",
message=f"Tool '{name}' called {count}x (threshold: "
f"{self.config['loop_threshold']})",
))
# 2. Check cost
total_cost = sum(s.get("cost_usd", 0) for s in spans)
if total_cost > self.config["max_cost_usd"]:
anomalies.append(Anomaly(
severity="warning",
category="cost",
message=f"Trace cost ${total_cost:.4f} exceeds "
f"${self.config['max_cost_usd']} threshold",
))
# 3. Check for token growth (context accumulation)
llm_spans = [s for s in spans if s.get("kind") == "llm"]
if len(llm_spans) >= 2:
first_input = llm_spans[0].get("input_tokens", 1)
last_input = llm_spans[-1].get("input_tokens", 1)
if first_input > 0:
ratio = last_input / first_input
if ratio > self.config["token_growth_ratio"]:
anomalies.append(Anomaly(
severity="warning",
category="cost",
message=f"Input tokens grew {ratio:.1f}x across "
f"{len(llm_spans)} turns (context bloat)",
))
# 4. Check for errors
error_spans = [s for s in spans if s.get("status") == "error"]
for es in error_spans:
anomalies.append(Anomaly(
severity="critical",
category="error",
message=f"Span '{es.get('name')}' failed: "
f"{es.get('error', 'unknown')}",
span_id=es.get("span_id"),
))
return anomalies
# Usage:
# analyzer = TraceAnalyzer()
# issues = analyzer.analyze(tracer.spans)
# for issue in issues:
# print(f"[{issue.severity}] {issue.category}: {issue.message}")
Comparing Good vs. Bad Traces
One of the most effective debugging techniques is side-by-side trace comparison. Take a trace where the agent succeeded and one where it failed for a similar query. Diff them systematically:
- Tool sequence: Did the failing trace call tools in a different order? Did it call extra tools? Did it skip a tool the good trace used?
- Tool arguments: Were the arguments to the same tool different? Was one set of arguments more specific?
- Token counts: Did the failing trace have significantly more input tokens at any turn? This suggests context pollution.
- Timing: Was any span in the failing trace unusually slow? Timeouts can cause cascading failures.
- Error propagation: Did an early error go unhandled, causing downstream failures?
Cost Debugging: "Why Did This Query Cost $2.50?"
When a single query costs significantly more than expected, the trace tells you exactly where the money went. Sort spans by cost (or by input_tokens + output_tokens if you haven't computed cost) and look at the top offenders. Common causes:
- Context accumulation: Each turn includes the full conversation history. By turn 15, you're sending 50,000 tokens per call. The trace shows input token counts growing linearly.
- Large tool results: A tool returns a 10,000-word document, which gets included verbatim in the next prompt. The trace shows the tool result size in the span metadata.
- Model escalation: Your router sent the query to Opus when Sonnet would have sufficed. The trace shows which model was used on each span.
- Retry storms: An intermittent error caused three retries, each with the full context. The trace shows duplicate spans with growing latency.
In a Cowork-style system with 30+ scheduled tasks, a task called daily_report_generator hit an empty database. Instead of failing gracefully, it entered a retry loop: search, get empty result, ask Claude "I didn't find anything, let me try differently," search again with a slightly different query, repeat. Each retry accumulated the prior conversation history. After 34 retries, the final turn sent 89,000 input tokens to Opus. The trace analyzer above would have caught this after the 3rd retry with a "loop" anomaly. Without tracing, it took two days to find.
From Debugging to Dashboards
Debugging is reactive. You look at traces after something goes wrong. Monitoring is proactive: you watch aggregate metrics to catch problems before users report them. Production agent monitoring requires a specific set of metrics that traditional web service monitoring doesn't cover.
The Golden Signals for Agent Systems
Google's Site Reliability Engineering book defines four golden signals: latency, traffic, errors, and saturation. For LLM agents, we need to extend these with agent-specific signals:
| Signal | Metric | Target | Alert Threshold |
|---|---|---|---|
| Latency | p50, p95, p99 response time | p50 < 3s, p99 < 15s | p99 > 30s for 5 min |
| Error Rate | Failed traces / total traces | < 2% | > 5% for 10 min |
| Cost per Query | Average USD per trace | $0.01 - $0.05 | > $0.20 average over 1 hour |
| Token Efficiency | Output tokens / input tokens ratio | 0.1 - 0.5 | Ratio > 1.0 (unusual verbosity) |
| Tool Success Rate | Successful tool calls / total | > 95% | < 85% for 15 min |
| Turns per Query | Average LLM calls per trace | 2 - 5 | > 10 average over 1 hour |
| Cache Hit Rate | Prompt cache hits / total calls | > 40% | < 20% (cache invalidated?) |
| Loop Detection | Traces with >3 same-tool calls | 0 | Any occurrence |
| Max Tokens Hit | % of calls hitting max_tokens | < 5% | > 15% (responses being truncated) |
Building a Monitoring Class
Here's a production-ready metrics collector that aggregates trace data into the golden signals above:
import time
import statistics
from collections import defaultdict, deque
from dataclasses import dataclass, field
from typing import Callable
@dataclass
class AlertRule:
name: str
check: Callable[["AgentMonitor"], bool]
message: str
cooldown_sec: int = 300
last_fired: float = 0
class AgentMonitor:
def __init__(self, window_size: int = 100):
self.latencies: deque = deque(maxlen=window_size)
self.costs: deque = deque(maxlen=window_size)
self.errors: deque = deque(maxlen=window_size)
self.turns_per_query: deque = deque(maxlen=window_size)
self.tool_results: deque = deque(maxlen=window_size * 5)
self.total_traces = 0
self.alert_handlers: list[Callable] = []
self.rules: list[AlertRule] = self._default_rules()
def record_trace(self, spans: list[dict]):
"""Record metrics from a completed trace."""
self.total_traces += 1
root = next((s for s in spans if not s.get("parent_id")), None)
if root:
self.latencies.append(root.get("latency_ms", 0))
self.errors.append(1 if root.get("status") == "error" else 0)
cost = sum(s.get("cost_usd", 0) for s in spans)
self.costs.append(cost)
llm_turns = sum(1 for s in spans if s.get("kind") == "llm")
self.turns_per_query.append(llm_turns)
for s in spans:
if s.get("kind") == "tool":
self.tool_results.append(s.get("status", "ok"))
self._check_alerts()
def get_dashboard(self) -> dict:
"""Return current metrics snapshot for a dashboard."""
lat = sorted(self.latencies) if self.latencies else [0]
return {
"total_traces": self.total_traces,
"p50_latency_ms": lat[len(lat) // 2],
"p99_latency_ms": lat[int(len(lat) * 0.99)],
"error_rate": sum(self.errors) / max(len(self.errors), 1),
"avg_cost_usd": statistics.mean(self.costs) if self.costs else 0,
"avg_turns": statistics.mean(self.turns_per_query)
if self.turns_per_query else 0,
"tool_success_rate": (
sum(1 for r in self.tool_results if r == "ok")
/ max(len(self.tool_results), 1)
),
}
def _default_rules(self) -> list[AlertRule]:
return [
AlertRule(
name="high_error_rate",
check=lambda m: (
sum(m.errors) / max(len(m.errors), 1) > 0.05
),
message="Error rate exceeds 5%",
),
AlertRule(
name="cost_spike",
check=lambda m: (
statistics.mean(m.costs) > 0.20 if m.costs else False
),
message="Average cost per query exceeds $0.20",
),
]
def _check_alerts(self):
now = time.time()
for rule in self.rules:
if (now - rule.last_fired > rule.cooldown_sec
and rule.check(self)):
rule.last_fired = now
for handler in self.alert_handlers:
handler(rule.name, rule.message)
SLOs for Agent Systems
Service Level Objectives for agents are tricky because "correct" is harder to define than for traditional APIs. A REST endpoint either returns the right data or it doesn't. An agent might return a response that's 80% correct, or correct but unhelpfully verbose, or technically correct but misses the user's intent.
Practical SLOs for agent systems:
- Availability SLO: 99.5% of agent invocations complete without an unhandled exception. This is your floor. If the agent crashes, that's always a failure.
- Latency SLO: 95% of queries complete in under 10 seconds, 99% in under 30 seconds. Adjust based on your use case. Interactive chat needs p99 < 10s. Batch processing can tolerate minutes.
- Cost SLO: 99% of queries cost less than $0.50. The 1% allows for complex multi-tool queries. Anything exceeding $0.50 should be investigated.
- Quality SLO (the hard one): This requires user feedback integration. Set a target like "90% of rated responses receive positive feedback." Track this with trace-correlated thumbs-up/thumbs-down signals.
User Feedback Integration
The most valuable signal in agent monitoring is user feedback, but only if it's correlated with traces. When a user clicks thumbs-down, you need to instantly link that feedback to the specific trace that generated the response. This lets you:
- Pull up the exact trace and see what went wrong
- Aggregate negative feedback by tool, model, or query type
- Identify systematic issues (e.g., "all queries involving the calendar tool get negative feedback")
- Build evaluation datasets from real failures
Implementation is straightforward: when generating a response, attach the trace_id to the response payload. When the user submits feedback, log it with the same trace_id. Your dashboard can then join these datasets.
Production Alerting Channels
Your monitoring system should alert through multiple channels based on severity:
- Critical (P1): Error rate > 10%, complete agent downtime, or a single trace costing > $5. Route to PagerDuty / on-call phone.
- Warning (P2): Error rate > 5%, cost spike > 2x baseline, latency p99 > 30s. Route to Slack #agent-alerts channel.
- Info (P3): New failure patterns, cache hit rate drop, model routing anomalies. Route to daily digest email.
A production learning system running multiple agent types needs distinct monitoring per agent role. The research agent has a higher acceptable latency (30s) but lower cost tolerance ($0.10). The chat agent has strict latency requirements (5s p95) but can use cheaper models. The evaluation agent runs in batch mode and cares most about consistency. Each role gets its own SLO targets and alert thresholds, all fed from the same trace data.
Tracing as a Cost Reduction Tool
Tracing isn't just for debugging. It's your most powerful tool for cost optimization. Without per-query cost visibility, you're optimizing blind. With it, you can identify the 20% of queries consuming 80% of your budget and target them precisely.
Understanding Model Costs
First, internalize the cost landscape. Prices as of mid-2026 (always verify current pricing):
| Model | Input (per 1M tokens) | Output (per 1M tokens) | Relative Cost | Typical Use |
|---|---|---|---|---|
| Claude Opus 4 | $15.00 | $75.00 | 1x (baseline) | Complex reasoning, code generation |
| Claude Sonnet 4 | $3.00 | $15.00 | 0.2x | General purpose, tool use |
| Claude Haiku 4 | $0.80 | $4.00 | 0.05x | Classification, simple tasks |
| GPT-4o | $2.50 | $10.00 | 0.15x | General purpose alternative |
| GPT-4o-mini | $0.15 | $0.60 | 0.01x | High-volume, simple tasks |
The cost difference between Haiku and Opus is 20x. That means routing even 30% of your traffic from Opus to Haiku could reduce your bill by 50%. But you can only identify which queries are Haiku-eligible if you have trace data showing query complexity.
Identifying Expensive Traces
The first step in cost optimization is understanding your cost distribution. Build a cost analyzer that ranks traces by expense and identifies patterns:
import statistics
from collections import Counter, defaultdict
from typing import Any
MODEL_COSTS_PER_1K = {
"claude-opus-4-20250514": (0.015, 0.075),
"claude-sonnet-4-20250514": (0.003, 0.015),
"claude-haiku-4-20250414": (0.0008, 0.004),
}
class CostAnalyzer:
def __init__(self, traces: list[list[dict]]):
"""Each trace is a list of spans."""
self.traces = traces
def cost_of_span(self, span: dict) -> float:
model = span.get("model", "")
inp = span.get("input_tokens", 0)
out = span.get("output_tokens", 0)
rates = MODEL_COSTS_PER_1K.get(model, (0, 0))
return (inp / 1000) * rates[0] + (out / 1000) * rates[1]
def top_expensive_traces(self, n: int = 10) -> list[dict]:
ranked = []
for trace in self.traces:
total = sum(self.cost_of_span(s) for s in trace)
trace_id = trace[0].get("trace_id", "?") if trace else "?"
ranked.append({"trace_id": trace_id, "cost": total,
"spans": len(trace)})
ranked.sort(key=lambda x: x["cost"], reverse=True)
return ranked[:n]
def cost_by_model(self) -> dict[str, float]:
by_model: dict[str, float] = defaultdict(float)
for trace in self.traces:
for span in trace:
model = span.get("model")
if model:
by_model[model] += self.cost_of_span(span)
return dict(by_model)
def model_routing_opportunities(self) -> list[dict]:
"""Find traces using expensive models that might work with cheaper ones.
Heuristic: short input, low output, single tool call = probably Haiku-eligible.
"""
opportunities = []
for trace in self.traces:
llm_spans = [s for s in trace if s.get("kind") == "llm"]
for span in llm_spans:
model = span.get("model", "")
inp = span.get("input_tokens", 0)
out = span.get("output_tokens", 0)
if ("opus" in model or "sonnet" in model) \
and inp < 2000 and out < 500:
savings = self.cost_of_span(span) - (
(inp / 1000) * 0.0008 + (out / 1000) * 0.004
)
opportunities.append({
"span_id": span.get("span_id"),
"current_model": model,
"suggested": "claude-haiku-4-20250414",
"potential_savings": round(savings, 6),
})
return opportunities
Prompt Caching Strategies
Anthropic's prompt caching can reduce input token costs by up to 90% for repeated prefixes. Tracing tells you whether caching is actually working:
- Track cache_creation_input_tokens and cache_read_input_tokens in your spans. The Anthropic API returns these in the usage object. If
cache_read_input_tokensis zero, your cache isn't being hit. - Common cache miss causes: System prompt changed between calls (even a single character invalidates the cache), conversation history order changed, tools list modified.
- Optimization: Keep your system prompt and tools definition stable. Put the static parts first (they form the cache prefix). Put dynamic content (user message, conversation history) last.
In your traces, add a computed field: cache_hit_rate = cache_read_input_tokens / (cache_read_input_tokens + input_tokens). Dashboard this over time. A sudden drop means something invalidated your cache.
Token Waste Analysis
Common sources of token waste, identifiable through traces:
- Over-long system prompts: Your system prompt is 3,000 tokens but most of it is rarely relevant. Trace analysis can show which parts of the system prompt are actually referenced in responses (by comparing system prompt content with response content). Trim the unused parts.
- Unnecessary tool calls: The agent calls
get_current_timeon every query, even when time is irrelevant. Trace analytics showing tool call frequency per query type reveal this pattern. - Redundant context: The agent includes full search results when a summary would suffice. Traces show the tool result size vs. how much of it appears in the final response.
- Conversation history bloat: In multi-turn conversations, each turn includes all previous turns. Traces show input token counts growing linearly per turn. Solutions: summarize older turns, use a sliding window, or use prompt caching aggressively.
A/B Testing Model Versions
Traces enable rigorous A/B testing of model versions. Route 10% of traffic to a new model, capture traces from both, and compare:
- Quality: Correlate user feedback scores with model version
- Cost: Compare average cost per trace between versions
- Latency: Compare p50/p95/p99 latency
- Tool use patterns: Does the new model use tools differently? More efficiently?
- Error rates: Does the new model fail more often?
In a multi-model setup where you route between Opus, Sonnet, and Haiku based on query complexity, traces give you the ground truth for your routing decisions. If your router sends 40% of traffic to Opus but trace analysis shows 60% of those queries are simple enough for Sonnet, you've found a significant cost optimization.
In most agent systems, 80% of cost comes from 20% of queries. These are typically queries that trigger multiple tool calls, use expensive models, or get stuck in retry loops. Trace your top 20% by cost and optimize those specifically. The remaining 80% of queries are usually cheap enough that optimizing them yields diminishing returns.
Reference Architecture
Let's bring everything together. Here's a reference architecture for a production agent system with full observability, modeled after a Cowork-style setup with 30+ scheduled tasks, a command dispatcher, and multi-model routing:
Choosing the Right Tool for Your Scale
Here's a decision tree to guide your tool selection:
Self-Hosted vs. SaaS: The Tradeoffs
This decision deserves careful consideration because it affects cost, maintenance burden, and data governance:
- SaaS (LangSmith, Braintrust, Helicone cloud): Zero infrastructure maintenance. Automatic updates. Shared community features. But: your prompt data leaves your infrastructure, which may violate compliance requirements. Costs scale linearly with volume. You're dependent on the vendor's uptime and roadmap.
- Self-Hosted (Phoenix, OTel + Tempo, Helicone self-hosted): Full data control. Fixed infrastructure cost (doesn't scale with query volume once provisioned). You can customize everything. But: you own the ops burden. Upgrades, scaling, backups, and monitoring the monitoring system are all your responsibility.
My recommendation: start with SaaS to move fast, build a thin abstraction layer, and migrate to self-hosted when you hit one of these triggers: (a) your monthly SaaS bill exceeds the cost of a dedicated VM, (b) a compliance requirement demands data residency, or (c) you need customization the SaaS doesn't support.
OpenTelemetry as the Universal Standard
Regardless of which tool you choose, instrument with OpenTelemetry semantics. Here's why:
- Portability: OTel traces can be exported to any compatible backend. Switch from Phoenix to Grafana Tempo to Datadog without changing your application code.
- Ecosystem: OTel has auto-instrumentation libraries for HTTP clients, databases, and message queues. Your agent's non-LLM dependencies get traced for free.
- Standards: The GenAI semantic conventions are being standardized. Early adoption means your traces will be compatible with future tooling.
- Correlation: OTel lets you correlate LLM traces with infrastructure traces. An agent that's slow because the database is slow shows up as one connected trace, not two disconnected investigations.
Retention Policies
Traces are verbose. A single agent query might generate 5-20KB of trace data. At 10,000 queries/day, that's 50-200MB/day, or 1.5-6GB/month. Manageable, but it adds up.
Recommended retention strategy:
- Hot storage (full detail, 7 days): Keep all spans, all metadata, all input/output text. This is your debugging window.
- Warm storage (aggregated, 90 days): Keep span trees and metadata, but drop full prompt/completion text. Replace with summaries or hashes. This is your trend analysis window.
- Cold storage (metrics only, 1 year): Keep only aggregated metrics: cost, latency, error counts, model usage. This is your capacity planning window.
- Special retention: Keep error traces and traces with negative user feedback at full detail indefinitely. These are your most valuable debugging and training data.
Building Custom Dashboards
If you're using Grafana (which you should, because it's free and excellent), here are the panels for your agent observability dashboard:
- Top row: Four stat panels showing current error rate, p50 latency, average cost per query, and total traces in the last hour
- Second row: Time series of error rate and latency (p50, p95, p99) over the last 24 hours
- Third row: Bar chart of cost by model, pie chart of tool usage distribution
- Fourth row: Table of recent error traces (linked to trace viewer), table of most expensive traces
- Bottom row: Cache hit rate over time, turns-per-query distribution histogram
The Meta-Problem: Observability of Your Observability
There's a delicious irony in building observability for AI agents: the observability system itself needs monitoring. If your trace collector goes down, you lose visibility at the exact moment you need it most (because something is probably going wrong).
Practical measures:
- Health checks: Monitor your trace collector's uptime independently of the traces it collects. A simple HTTP health check endpoint, monitored by an external service (UptimeRobot, Pingdom), is sufficient.
- Queue depth monitoring: If you're using a batch span processor (and you should be, for performance), monitor the queue depth. A growing queue means you're producing traces faster than you can export them.
- Sampling fallback: If trace export fails, don't lose the data silently. Fall back to writing spans to a local file that can be replayed later. Better to have a big log file than no traces.
- Separate critical path: Your trace export should never block your agent's response path. Use async export with a bounded queue. If the queue is full, drop spans (with a counter metric), don't slow down the agent.
Putting It All Together: Your First Week
If you're starting from zero, here's a practical one-week plan to go from no observability to a production-grade setup:
- Day 1: Add the basic trace decorator (Module 4, Pattern 1) to your agent's LLM calls. Deploy. You now have latency and cost visibility.
- Day 2: Add tool call tracing (Pattern 2). Deploy. You now have full span trees.
- Day 3: Set up a trace backend. If SaaS: sign up for Braintrust or LangSmith, add their SDK. If self-hosted:
pip install arize-phoenixandphoenix serve. Export your spans. - Day 4: Build the trace analyzer (Module 5). Add it as a post-processing step. Log anomalies to your team's Slack channel.
- Day 5: Set up the monitoring class (Module 6). Create a Grafana dashboard (or use your trace tool's built-in dashboard) with the golden signals.
- Day 6: Run the cost analyzer (Module 7) on your first week of data. Identify the top 10 most expensive traces. Fix the obvious wins (loops, model routing).
- Day 7: Write documentation for your team. Add alerting rules. Celebrate: you've gone from black box to glass box in one week.
The best observability system is the one you actually use. Don't over-engineer your first version. Start with the basic decorator, iterate based on what you learn from the data, and invest in more sophisticated tooling only when the data tells you where to look. Your agents are already running. Start watching them today.
Need this for a date?
Turn this course into a ramp-up pack sized to your minutes per day, or build an interview or certification pack for the day you need it.