How to trace and debug AI agent decisions

Tracing an agent means giving each run one trace ID and recording every step it takes (each model call, tool call and decision) as a span linked to its parent, so you can follow one run from start to finish and replay the exact inputs behind a bad decision.

This guide uses the terms as OpenTelemetry, an open-source framework for generating and exporting traces, metrics and logs, defines them: a trace is the path of one request through your system, and a span is one unit of work within it.

Anyone with an agent in production who has been asked “why did it do that?”: a support bot that refunded the wrong order, a coding agent that deleted a file, a pipeline that filed a document under the wrong customer. And anyone whose agent runs across several services or background jobs, where the logs from one run end up scattered.

Agent decisions are hard to reproduce. The same prompt can produce a different answer, the context the model saw was put together at runtime from tools and retrieval, and the run may have crossed several services. Plain logs show that something happened, but not which model call, with which input, led to which action, in which run.

1. One trace ID per run. Every span in the run carries it. The W3C Trace Context standard makes a trace ID 16 bytes, written as 32 lowercase hex characters, and a span ID 8 bytes (16 characters); an ID of all zeros is invalid.
2. One span per step. A span for the run, and one for each model call and tool call, each pointing at its parent. OpenTelemetry’s conventions for generative AI name these spans by operation, such as chat {model}, execute_tool {tool} and invoke_agent {agent}, and define attributes such as gen_ai.request.model and gen_ai.usage.input_tokens.
3. Keep the exact inputs and outputs, but not on the span. The same conventions treat prompts and outputs as sensitive and often large, say instrumentation shouldn’t capture them by default, and recommend storing them externally with a reference on the span in production when volume or sensitive data is a concern. That separate store is what makes replay possible.
4. Carry the trace across services. When the run calls another service or queues a job, pass the trace context along, for example in a W3C traceparent header.

/*
 * A minimal tracer with the same shape as OpenTelemetry: one trace ID per
 * agent run, one span per step. Prompts and outputs go to a separate store
 * and the span keeps a reference, so they can be replayed later.
 */
export interface Span {
  traceId: string // 32 lowercase hex characters
  spanId: string // 16 lowercase hex characters
  parentSpanId?: string
  name: string
  attributes: Record<string, string | number>
}

export interface ModelCall {
  provider: string
  model: string
  input: unknown // the exact messages sent
  output: unknown // the exact response received
  inputTokens: number
  outputTokens: number
}

function randomHex(bytes: number): string {
  const buf = crypto.getRandomValues(new Uint8Array(bytes))
  const hex = Array.from(buf, (b) => b.toString(16).padStart(2, '0')).join('')
  return /^0+$/.test(hex) ? randomHex(bytes) : hex // all zeros is invalid
}

export function createTracer() {
  const spans: Span[] = []
  const content = new Map<string, ModelCall>() // keep this store private

  function startSpan(name: string, parent?: Span): Span {
    const span: Span = {
      traceId: parent?.traceId ?? randomHex(16),
      spanId: randomHex(8),
      parentSpanId: parent?.spanId,
      name,
      attributes: {},
    }
    spans.push(span)
    return span
  }

  function recordModelCall(parent: Span, call: ModelCall): Span {
    const span = startSpan(`chat ${call.model}`, parent)
    span.attributes = {
      'gen_ai.operation.name': 'chat',
      'gen_ai.provider.name': call.provider,
      'gen_ai.request.model': call.model,
      'gen_ai.usage.input_tokens': call.inputTokens,
      'gen_ai.usage.output_tokens': call.outputTokens,
      'app.content_ref': span.spanId, // the prompt itself is not on the span
    }
    content.set(span.spanId, call)
    return span
  }

  /** Every model call in a trace, in order, with what went in and out. */
  function replay(traceId: string): ModelCall[] {
    return spans
      .filter((s) => s.traceId === traceId && content.has(s.spanId))
      .map((s) => content.get(s.spanId)!)
  }

  /** W3C traceparent header, to carry the trace to another service. */
  const traceparent = (span: Span) => `00-${span.traceId}-${span.spanId}-01`

  return { spans, startSpan, recordModelCall, replay, traceparent }
}

The example has no dependencies so the shape is easy to see. In a real app, use an OpenTelemetry SDK instead of hand-made IDs, and keep the separate content store and the reference attribute. The GenAI conventions are marked “Development”, so attribute names can still change; check the current version before you build dashboards on them.

Generated code often logs a line before and after a model call. What it rarely adds unless asked is the structure that makes those lines useful later: an ID shared by every step of a run, links from each step to its parent, and the exact model input stored somewhere you can query. Without them, debugging a bad decision starts with guessing which log lines belong together.

Replays also give your coding agent something concrete to work with: a stored input that reproduces a bad decision becomes a regression test.

• Plain logs with a run ID. Cheap and much better than nothing. No parent links or timings, so you rebuild the tree of steps by hand.
• OpenTelemetry. Vendor- and tool-agnostic, with SDKs for many languages, and context propagation ties spans from different services into one trace. Its GenAI conventions are still in development.
• Hosted LLM tracing tools and framework tracing. They save setup. Check whether they accept OpenTelemetry data, where they store prompts and who can read them.
• Everything on the span. The simplest way to keep prompts, but the GenAI conventions warn that this content is likely to contain sensitive data, including personal data, and can exceed the size limits observability backends set for attribute values.

1. Create a trace ID when a run starts and pass it to every step, job and service the run touches.
2. Wrap each model call and tool call in a span with the GenAI attributes: operation, provider, model and token counts.
3. Store the exact messages and responses in a separate store with its own access control, and put the reference on the span.
4. Write a replay script: given a trace ID, load the stored inputs and run them again against the same model or a new one.
5. Turn every bad decision you replay into a test.

What to tell your coding agent. Paste this into Claude Code, Codex or Antigravity:

Add tracing to every agent run in this project. Create a trace ID when a run starts and pass it through every model call, tool call, background job and service the run touches (use the W3C traceparent header between services). Record each model call and tool call as a span with a parent, timings and the OpenTelemetry GenAI attributes gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. Store the exact model inputs and outputs in a separate access-controlled store keyed by span ID, and put only that reference on the span. Add a script that replays every model call in a trace from the stored inputs.

To make it a standing rule instead of a one-off request, put the same text in the instructions file your agent reads at the start of a session: CLAUDE.md for Claude Code, AGENTS.md for Codex, and AGENTS.md, GEMINI.md or a file in .agents/rules/ for Antigravity. ArcLab doesn’t connect to any of these tools; you paste the text yourself.

The Production Observability, Tracing & Decision Replay module in the Integrated Systems track starts from an example incident, an automated portfolio manager that sold a customer’s assets without warning. It covers carrying trace context across services, span attributes you can query, a replay harness and alert thresholds, and ends with an observability and tracing spec for your own system.

The official documentation this guide was checked against:

• What is OpenTelemetry?, OpenTelemetry docs: vendor- and tool-agnostic; language SDKs
• Traces, OpenTelemetry docs: traces, spans, span context, attributes, propagation
• Semantic conventions for generative client AI spans, OpenTelemetry GenAI semantic conventions: execute_tool spans, capturing content, Development status
• Semantic conventions for generative AI client inference, OpenTelemetry GenAI semantic conventions: chat span names; gen_ai.* attributes and token counts
• Semantic conventions for GenAI agent and framework spans, OpenTelemetry GenAI semantic conventions: invoke_agent spans
• Trace Context, W3C Recommendation: traceparent format and ID sizes
• How Claude remembers your project, Claude Code docs, Anthropic: CLAUDE.md is read at the start of every session
• AGENTS.md, Codex docs, OpenAI: Codex reads AGENTS.md before doing any work
• Rules, Google Antigravity docs: workspace rules in AGENTS.md, GEMINI.md or .agents/rules/