Production Agentic Observability & OpenTelemetry Tracing

Operating autonomous agent workflows at enterprise scale requires real-time observability across the entire call graph. When an agent fails or stalls, engineering teams cannot afford to sift through 200MB text logs. They require structured, distributed OpenTelemetry (OTel) traces capturing parent-child span hierarchies, token billing consumption, tool execution latencies, and microVM exit codes.

OpenTelemetry Semantic Conventions for GenAI

Production agents must adhere to the OpenTelemetry semantic conventions for generative AI. Every discrete LLM invocation and tool call generates a child span linked to the parent task root:

• gen_ai.system: Anthropic / OpenAI / Google / DeepSeek
• gen_ai.request.model: claude-3-7-sonnet / gpt-4-5
• gen_ai.usage.input_tokens: Total prompt tokens ingested
• gen_ai.usage.output_tokens: Tokens emitted
• gen_ai.usage.cached_tokens: KV cache hit tokens
• agent.tool.name: Executed tool identifier (e.g., bash_microvm_exec)
• agent.tool.status: success or error

Instrumenting Agent Execution with Python OpenTelemetry SDK

Here is a reference implementation of a context-managed trace span wrapper:

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("xspy.agentic.orchestrator")

async def execute_traced_tool(tool_name: str, args: dict, parent_span=None):
    with tracer.start_as_current_span(f"tool_exec:{tool_name}") as span:
        span.set_attribute("agent.tool.name", tool_name)
        span.set_attribute("agent.tool.arg_count", len(args))
        
        try:
            result = await run_tool_in_sandbox(tool_name, args)
            span.set_attribute("agent.tool.status", "success")
            span.set_status(Status(StatusCode.OK))
            return result
        except Exception as e:
            span.record_exception(e)
            span.set_attribute("agent.tool.status", "error")
            span.set_status(Status(StatusCode.ERROR, str(e)))
            raise

Token Budget Leak Detection in CI/CD

In automated regression pipelines, trace spans are analyzed against anomalous token inflation. If a prompt refactor causes an agent's recursive tool calling to spike from 15,000 tokens to 120,000 tokens per task, the OpenTelemetry alert fires immediately, halting pull request merges before downstream API bills compound.