How an LLM Uses Tools: The Step-by-Step Anatomy of a Function Call and JSON-RPC Handshake

Popular media routinely describes autonomous AI agents as "browsing the web," "executing SQL queries," or "ordering supplies." These anthropomorphic metaphors obscure a fundamental architectural reality:

Tool calling is fundamentally a four-phase distributed communication protocol coordinated through structured schemas and strict state-machine handshakes.

1. The Four-Phase Execution Lifecycle

To understand tool calling from first principles, we trace the full round-trip journey of a user asking an agent: "Query the production cluster status and summarize any failing pods."

[1. User Query + Tools Schema] ===> [Frontier LLM]
                                           |
                                    (Detects missing live data)
                                           v
[2. Client Intercepts JSON Call] <=== [Emits `tool_calls` JSON]
         |
    (Runs sandbox command: `kubectl get pods -n prod`)
         v
[3. Tool Observation Return] ====> [Frontier LLM]
                                           |
                                    (Synthesizes answer)
                                           v
[4. Final Markdown Report]   <==== [Emits Assistant Message]

2. Phase 1: JSON Schema Injection into System Prompts

Before the model can invoke a tool, it must be informed of the tool's signature, expected parameter types, and semantic purpose. This is achieved by injecting tool definitions formatted in JSON Schema into the hidden system prompt payload:

{
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "execute_kubernetes_query",
        "description": "Inspect live cluster pod health, restarts, and deployment status.",
        "parameters": {
          "type": "object",
          "properties": {
            "namespace": { 
              "type": "string", 
              "description": "Target Kubernetes namespace (e.g. 'prod', 'staging')." 
            },
            "status_filter": { 
              "type": "string", 
              "enum": ["CrashLoopBackOff", "Failed", "All"],
              "description": "Filter pods by lifecycle status." 
            }
          },
          "required": ["namespace"]
        }
      }
    }
  ]
}

Modern fine-tuned models (such as Claude 3.7, GPT-4o, or Qwen 2.5) are heavily trained on synthetic dataset trajectories to recognize that whenever a user prompt requires external facts, they must suspend natural prose generation and emit structured function calls.

3. Phase 2: Constrained Grammar Emission & Stop Sequences

In early implementations (2022), models frequently emitted malformed JSON strings (e.g., trailing commas, unescaped quotes, missing brackets) that crashed standard parsers. Modern serving engines (such as vLLM and SGLang) enforce Grammar-Constrained Decoding:

The model outputs special protocol tokens (e.g. <tool_call>) followed by the arguments payload, and immediately terminates its turn with finish_reason: "tool_calls":

{
  "role": "assistant",
  "content": null,
  "tool_calls": [
    {
      "id": "call_k8s_98412_prod",
      "type": "function",
      "function": {
        "name": "execute_kubernetes_query",
        "arguments": "{\"namespace\": \"prod\", \"status_filter\": \"CrashLoopBackOff\"}"
      }
    }
  ]
}

4. Phase 3: Client Sandboxing, Argument Validation & Execution

Upon receiving finish_reason: "tool_calls", the client application intercepts the response. Before executing any local command, a production agent harness enforces three critical guardrails:

1. Schema Validation: Parses the JSON arguments against a strongly-typed schema validator (Pydantic in Python, Zod in TypeScript) to guarantee type safety.
2. Human-in-the-Loop Interception: For destructive operations (e.g. DROP TABLE, git push --force, financial transactions), the client pauses and prompts the user for manual approval.
3. Sandbox Isolation: Executes shell commands or python scripts inside isolated MicroVMs (Firecracker) or Linux containers with drop-capabilities (Bubblewrap) to prevent host compromise.

Resilient Error Handling Protocol

If the external database or API returns an error (e.g., HTTP 404 Not Found or AuthenticationFailure), the client must not crash. Instead, it serializes the error message into a structured observation payload and submits it back to the model. The model reads the error and autonomously self-corrects (e.g., re-trying with corrected parameters).

5. Phase 4: Observation Re-Injection & Final Synthesis

The client appends a new message with role: "tool" containing the execution result, referencing the exact tool_call_id emitted by the model:

{
  "role": "tool",
  "tool_call_id": "call_k8s_98412_prod",
  "name": "execute_kubernetes_query",
  "content": "{\"failing_pods\": [{\"pod_name\": \"payment-service-84f9\", \"restarts\": 14, \"reason\": \"OOMKilled\"}]}"
}

The client then sends the complete conversation history (User Query + Assistant Tool Call + Tool Observation) back to the LLM. Now possessing the verified facts in its context window, the model generates the final human-readable response:

{
  "role": "assistant",
  "content": "The production cluster reports **1 failing pod** in namespace `prod`:\\n\\n- **payment-service-84f9**: Trapped in `CrashLoopBackOff` with 14 restarts due to an **OOMKilled** memory limit breach."
}

6. The Model Context Protocol (MCP) Standard

Until recently, every AI provider used proprietary, incompatible function-calling formats. Anthropic introduced the Model Context Protocol (MCP) to establish an open standard modeled on the Language Server Protocol (LSP).

Under MCP, tool providers run standalone servers communicating via JSON-RPC 2.0 over standard input/output (stdio) or Server-Sent Events (SSE):

• tools/list: Client discovers available tools dynamically at runtime without hardcoding.
• tools/call: Client issues execution request with standardized JSON-RPC parameters.
• resources/read: Provides structured access to internal files, database schemas, and documentation.

7. Resilient Production Implementation

The following Python script implements a self-healing tool loop that validates schemas and catches API exceptions:

import json
from typing import Dict, Any

def execute_safe_tool(tool_call: Dict[str, Any]) -> Dict[str, Any]:
    func_name = tool_call["function"]["name"]
    raw_args = tool_call["function"]["arguments"]
    call_id = tool_call["id"]

    try:
        args = json.loads(raw_args)
        
        # Dispatch to verified local implementation
        if func_name == "get_system_telemetry":
            result = {"status": "HEALTHY", "cpu_percent": 14.2, "memory_free_gb": 48.6}
        else:
            raise ValueError(f"Unknown tool function '{func_name}'")

        return {
            "role": "tool",
            "tool_call_id": call_id,
            "name": func_name,
            "content": json.dumps(result)
        }
    except Exception as exc:
        # Return structured error back to LLM for autonomous self-correction
        return {
            "role": "tool",
            "tool_call_id": call_id,
            "name": func_name,
            "content": json.dumps({"error": str(exc), "retry_instruction": "Verify parameter schema."})
        }