How agents actually use tools

An AI agent does not literally reach into your computer and press a button or execute a command. What happens behind the scenes is much more disciplined: an orchestrated loop of schema serialization, token generation, structured parsing, execution in a sandboxed host environment, and stateful conversational injection.

Understanding this mechanism transforms “agentic AI” from what sounds like science fiction into a tangible, inspectable distributed systems problem.

The Mental Model: Models Produce Strings, Runtimes Act

At its fundamental level, a Large Language Model is a stateless function that maps a token sequence to a probability distribution over the next token:

$P(t_n \mid t_1, t_2, \dots, t_{n-1})$

The model cannot initiate network sockets, read from disks, or query databases on its own. Instead, tool execution relies on a cooperation between two entities:

1. The Reasoning Engine (The LLM): Analyzes conversation history and decides whether an action is required, which tool to invoke, and what arguments to pass.
2. The Execution Environment (The Runtime Host): Holds execution permissions, resolves network calls, runs code, and feeds results back to the model.

+----------------------------------------------------------------+
| Host Runtime (Python / Node / Go)                              |
|                                                                |
|  1. Injects Tool Schemas (JSON Schema) into Context            |
|  2. Calls Model API -----------------------+                   |
|                                            |                   |
|                                            v                   |
|                              +---------------------------+     |
|                              | Large Language Model      |     |
|                              | Emits structured tokens   |     |
|                              +---------------------------+     |
|                                            |                   |
|  3. Parses Tool Call <---------------------+                   |
|  4. Dispatches function (e.g. SQL, API, Shell)                 |
|  5. Injects observation back into conversation                 |
|  6. Prompts model to generate next response                    |
+----------------------------------------------------------------+

Step 1: Declaring Tool Schemas

Before the model can invoke a tool, it needs to know what tools exist and what parameters they require. Modern model APIs accomplish this by supplying schemas formatted in standard JSON Schema syntax.

Here is what a search tool definition looks like when transmitted to the inference engine:

search_tool_schema = {
    "type": "function",
    "function": {
        "name": "search_knowledge_base",
        "description": "Searches the internal documentation for architecture decisions and runbooks.",
        "parameters": {
            "type": "object",
            "properties": {
                "query": {
                    "type": "string",
                    "description": "Specific search query keywords"
                },
                "max_results": {
                    "type": "integer",
                    "description": "Maximum number of snippets to return (default: 5)",
                    "default": 5
                },
                "category": {
                    "type": "string",
                    "enum": ["architecture", "infrastructure", "incidents"],
                    "description": "Optional category filter"
                }
            },
            "required": ["query"]
        }
    }
}

Behind the scenes, the provider’s API translates this JSON schema into special tokens in the model’s system prompt or fine-tuned tool-calling format.

Step 2: Generation of Function Tokens

When a user asks:

“Why did our database cluster fail yesterday morning?”

The model evaluates its available tools. It recognizes that it lacks real-time knowledge of yesterday’s infrastructure events. Rather than hallucinating a response or emitting natural text, it switches modes and emits special delimiter tokens indicating a tool invocation:

{
  "role": "assistant",
  "content": null,
  "tool_calls": [
    {
      "id": "call_abc123",
      "type": "function",
      "function": {
        "name": "search_knowledge_base",
        "arguments": "{\"query\": \"database cluster outage incident\", \"category\": \"incidents\", \"max_results\": 3}"
      }
    }
  ]
}

Notice that the arguments are generated as a raw JSON string. The model is literally generating JSON character by character according to the grammar rules specified in the schema.

Step 3: Dispatch and Execution Loop

Once the runtime receives a response containing tool_calls, the application loop executes the local implementation.

A production dispatch loop handles routing, schema validation, timeout enforcement, and exception handling:

import json
import asyncio
from typing import Any, Dict

class ToolDispatcher:
    def __init__(self):
        self._registry = {}

    def register(self, name: str, fn):
        self._registry[name] = fn

    async def execute(self, tool_call: Dict[str, Any]) -> Dict[str, Any]:
        fn_name = tool_call["function"]["name"]
        raw_args = tool_call["function"]["arguments"]
        call_id = tool_call["id"]

        if fn_name not in self._registry:
            return {
                "role": "tool",
                "tool_call_id": call_id,
                "content": json.dumps({"error": f"Tool '{fn_name}' not found."})
            }

        try:
            parsed_args = json.loads(raw_args)
            # Execute with timeout safeguard
            result = await asyncio.wait_for(
                self._registry[fn_name](**parsed_args),
                timeout=15.0
            )
            return {
                "role": "tool",
                "tool_call_id": call_id,
                "content": json.dumps({"success": True, "data": result})
            }
        except json.JSONDecodeError:
            return {
                "role": "tool",
                "tool_call_id": call_id,
                "content": json.dumps({"error": "Failed to parse argument JSON from model."})
            }
        except asyncio.TimeoutError:
            return {
                "role": "tool",
                "tool_call_id": call_id,
                "content": json.dumps({"error": "Tool execution timed out after 15 seconds."})
            }
        except Exception as e:
            return {
                "role": "tool",
                "tool_call_id": call_id,
                "content": json.dumps({"error": str(e)})
            }

Step 4: Multi-Step Agentic Reasoning

Real-world problems rarely require just a single action. An agent often needs to decompose goals into sequential steps:

1. Search the logs for error spikes.
2. Identify the culprit service identifier (svc-auth-92).
3. Query the deployment ledger for recent commits to svc-auth.
4. Synthesize findings into an incident timeline.

Here is how tool patterns compare across architectural tiers:

Why Tool Calling Fails in Production

Building a demo agent with tool calling takes twenty minutes. Making it reliable in production reveals several non-obvious failure modes:

1. Schema Drift and Type Confusion

When schemas become too large or complex, models begin hallucinating enum options, formatting dates incorrectly, or swapping arguments between adjacent tools.

Rule of thumb: Keep tool signatures small and orthogonal. A tool with 15 optional parameters is far more likely to fail than three specialized tools with 2 parameters each.

2. Output Token Flooding

If your database query or log search returns 50,000 lines of JSON, injecting that entire payload into the context window will:

• Blow out your token budget.
• Introduce latency of several seconds.
• Dilute the model’s attention mechanism (often called “the needle in a haystack problem”).

Always truncate, summarize, or project tool outputs to only the specific keys the model needs.

3. Permission Escalation

An agent should never run with higher permissions than the authenticated user initiating the prompt. If user A cannot query financial payroll records, the database tool executed on behalf of user A must strictly enforce database-level row and column access controls.

Summary

Tool calling is fundamentally a protocol for structured RPC negotiation across a natural language interface. By treating it as a distributed systems interface rather than magic, you can design agents with predictable error boundaries, verifiable safety guarantees, and deterministic reliability.