DL S075 · Agents and Tool Use

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 75 / 80 · Module M12 · ~2 hours

The story we're starting with

Somewhere in 2022, a paper came out called ReAct — "Reasoning + Acting" — with an idea so embarrassingly simple that people spent a year not believing it worked. The idea: give the LLM a scratchpad, teach it to write down its next intended action as text, then when it writes the special string Action: calculator[2+2], pause generation, actually run the calculator, paste Observation: 4 back into its context, and let it keep generating. That's an agent. That's the whole thing. There is no separate "planning module", no reinforcement learning, no additional network. It's the same next-token predictor from S040 with a for-loop wrapped around it that watches for tool-call tokens.

The reason it works is a fact about pretrained LLMs that took the field a while to internalize: the LLM has already read millions of examples of humans thinking out loud and then acting. Blog posts, StackOverflow answers, tutorials, transcripts. The instinct "think, then act, then observe, then think again" is already in the weights. You don't need to teach it. You just need to (a) give it a syntax to signal "I want to act now", (b) intercept that signal and actually perform the action, and (c) hand back the result. The loop that wraps this up is under 60 lines of Python.

Today we build that loop from scratch, wire it to two tools (a Python REPL and a web-search stub), watch it succeed and fail on a few tasks, and then look at how the same idea got dressed up as "function calling" in the OpenAI API — same trick, different clothes.

Prerequisites

• S040 · nanoGPT rebuild. The agent loop is a wrapper around model.generate(). You should know what that is.
• S063 · Instruction fine-tuning. Function-calling models are instruction-tuned with a specific data mix; that recipe assumes you know what SFT looks like.
• S066 · Prompt engineering fundamentals. The system prompt is 80% of an agent's quality.
• Basic subprocess / eval familiarity in Python — we'll run untrusted-ish code and need to think about sandboxing.

1 · What an agent actually is

Strip all the marketing off "AI agent" and you're left with three things:

1. A prompt describing available tools and asking the model to solve a task.
2. A generation loop that stops when the model emits a tool-call marker.
3. A dispatcher that runs the tool, formats the result, and appends it back to the context.

There is nothing else. No separate planner, no memory subsystem, no RL. Every "agent framework" you'll ever see (LangChain, AutoGen, CrewAI, LlamaIndex agents) is a wrapper around these three things. Sometimes a bloated wrapper.

1.1 · The ReAct prompt format

The original ReAct paper used this scratchpad format:

You have access to the following tools:
- calculator(expression: str) -> str: Evaluates a Python arithmetic expression.
- search(query: str) -> str: Returns the top 3 search results for a query.
 
Use this format:
Thought: <your reasoning>
Action: <tool_name>[<arg>]
Observation: <will be filled in by the environment>
... (repeat Thought/Action/Observation as needed)
Thought: I know the final answer.
Final Answer: <answer>
 
Question: What is the population of France divided by the population of Germany?

The model then generates:

Thought: I need to look up both populations.
Action: search[population of France]

Your code sees Action: search[population of France], stops generation, runs the search, and injects:

Observation: France's population is approximately 68 million (2024).

Then resumes generation. The model continues:

Thought: Now I need Germany.
Action: search[population of Germany]

And so on until it emits Final Answer:.

2 · Building it from scratch

Let's write it. We'll assume an OpenAI-compatible chat endpoint (works with your local capstone model via vLLM or with a real API).

2.1 · The tool registry

import re, subprocess
 
def tool_calculator(expr: str) -> str:
    try:
        # NOT for production. Use ast.literal_eval or a real sandbox.
        result = eval(expr, {"__builtins__": {}}, {})
        return str(result)
    except Exception as e:
        return f"Error: {e}"
 
def tool_search(query: str) -> str:
    # Stub. In production this would hit SerpAPI, Brave, Tavily, etc.
    fake_db = {
        "population of france": "France's population is approximately 68 million (2024).",
        "population of germany": "Germany's population is approximately 84 million (2024).",
    }
    return fake_db.get(query.lower(), f"No results for '{query}'.")
 
TOOLS = {"calculator": tool_calculator, "search": tool_search}

Two tools. That's enough to see the loop work.

2.2 · The parser

The model will emit lines like Action: search[population of France]. Parse it:

ACTION_RE = re.compile(r"Action:\s*(\w+)\[(.+?)\]", re.DOTALL)
 
def parse_action(text: str):
    m = ACTION_RE.search(text)
    if not m:
        return None, None
    return m.group(1), m.group(2)

Robust enough for a demo. Production agents use JSON parsing (see §4).

2.3 · The loop

def run_agent(question: str, llm_call, max_steps: int = 8) -> str:
    prompt = SYSTEM_PROMPT + f"\nQuestion: {question}\n"
    trace = ""
    for step in range(max_steps):
        # Ask model to generate until it emits an Action or Final Answer.
        completion = llm_call(prompt + trace, stop=["Observation:", "Final Answer:"])
        trace += completion
 
        if "Final Answer:" in completion:
            # Get the rest after the marker.
            answer_completion = llm_call(prompt + trace, stop=None)
            trace += answer_completion
            return trace.split("Final Answer:")[-1].strip()
 
        tool_name, arg = parse_action(completion)
        if tool_name is None:
            trace += "\nObservation: [parse error — no action found]\n"
            continue
        if tool_name not in TOOLS:
            trace += f"\nObservation: [unknown tool '{tool_name}']\n"
            continue
 
        obs = TOOLS[tool_name](arg)
        trace += f"\nObservation: {obs}\n"
 
    return "[max steps exceeded]\n" + trace

That's the whole agent. ~30 lines including whitespace. Every framework you've ever heard of is a variation on this file.

2.4 · Trace of one run

For "population of France divided by Germany":

Thought: I need both populations.
Action: search[population of France]
Observation: France's population is approximately 68 million (2024).
Thought: Now Germany.
Action: search[population of Germany]
Observation: Germany's population is approximately 84 million (2024).
Thought: Now the division.
Action: calculator[68/84]
Observation: 0.8095238095238095
Final Answer: About 0.81 (France has roughly 81% of Germany's population).

Four LLM calls, two searches, one calculator hit. Total latency: whatever your model's generation speed is, dominated by the LLM. The "reasoning" is real but happens entirely in the text stream. There is no hidden state.

3 · Function calling — the same trick, dressed up

OpenAI shipped "function calling" in June 2023. It's ReAct with two differences:

1. JSON schema up front. You declare tools as JSON schemas, so the model knows the exact argument shape.
2. A special token boundary. Instead of parsing free text, the model outputs a structured object; the API decodes it for you.

Under the hood, function-calling models are just SFT'd on a data mix that includes examples like:

[user] What's the weather in Paris?
[assistant] {"tool_call": {"name": "get_weather", "args": {"city": "Paris"}}}
[tool result] {"temp": 18, "condition": "cloudy"}
[assistant] It's 18°C and cloudy in Paris right now.

That's it. The "function calling capability" is 100k SFT examples of this shape. There's no new architecture. If you fine-tuned your capstone model on the same shape of data, it would function-call too. (In fact, the open-source functionary and Mistral's Tool models did exactly that.)

3.1 · Why the JSON form is better in production

• Fewer parse errors. The model tends to emit valid JSON when SFT'd for it; free text Action: foo[bar] breaks on nested brackets or commas.
• Easier arg validation. You have a schema, so you can reject malformed calls at the framework layer instead of relying on the model.
• Streaming-friendly. The API can tell you "I'm now emitting a tool call" as soon as the first JSON token appears, without waiting for a specific string.

3.2 · Why the text form is fine for hobby projects

Everything above matters at scale. For your capstone: a regex over free text is 30 lines, works on any model, and requires zero SFT. Ship the simple thing first.

4 · Where agents fall over

5 · When you don't need an agent

The industry over-uses agents. If your task is:

• Single-turn Q&A: don't. Just prompt the model directly.
• Retrieval-then-answer: don't. Just RAG (see S076 tomorrow).
• Structured extraction: don't. Just use JSON-mode / grammar-constrained decoding.

You need an agent when the task is truly multi-step with data-dependent branching — e.g., "book the cheapest flight for a trip that satisfies these 5 constraints, checking calendar and budget as you go." That's an agent. "Summarize this document" is not.

6 · Diagram — the agent loop

7 · The 2024–2025 agent landscape — what changed

Between Yao et al.'s original ReAct paper (Oct 2022, arXiv:2210.03629) and today, agents went from research demo to a $B+ market. The vocabulary you need:

7.1 · Reflexion (Shinn et al., 2023, arXiv:2303.11366)

Standard ReAct has no memory across attempts — fail a task, retry from scratch, fail the same way. Reflexion adds a verbal self-critique step: after failure, the agent generates a natural-language reflection ("I searched for X but the tool returned Y — next time I should try Z first") and stores it. On the next attempt, the reflection is prepended to the prompt. On HumanEval coding, Reflexion pushed GPT-4 from ~67% to ~91% pass@1 without any weight updates. That's a 24-point jump from adding a paragraph of prompt context. Storage is the cheapest form of intelligence.

7.2 · Voyager (Wang et al., 2023, arXiv:2305.16291)

Voyager — a Minecraft-playing GPT-4 agent — introduced skill library growth: successful action sequences get compiled into named Python functions and added to a growing library the agent can call in future episodes. After ~160 episodes, Voyager had bootstrapped a library of ~200 skills (mine wood → craft table → craft pickaxe → mine stone → …) and unlocked diamond-tier items 3× faster than baseline ReAct. Lesson: the tool set should grow with experience, not be fixed at design time.

7.3 · Native tool-calling and structured decoding (2024)

OpenAI's tools=[...] parameter (Nov 2023) and Anthropic's tool-use API (Apr 2024) moved tool declarations out of the prompt into a first-class API field. Under the hood the model is still SFT'd on tool-formatted data — but the API layer handles the JSON grammar and parses arguments for you. Combined with constrained decoding libraries like Outlines (arXiv:2307.09702) and XGrammar, you can now guarantee the model outputs a valid tool call — no more "model hallucinated a trailing comma and my JSON parser died at 2am."

7.4 · Computer-use agents (late 2024 → 2025)

The biggest shift: agents that don't call APIs, they operate a mouse and keyboard on a screenshot. Anthropic's Computer Use (Claude 3.5 Sonnet, Oct 2024) and OpenAI's Operator / CUA (Jan 2025) take a screenshot, output pixel coordinates + click/type actions, take another screenshot, repeat. The tool interface is (screenshot) → (x, y, action). Same ReAct loop you built; the observation is a PNG instead of a string.

Why this matters: most enterprise software has no API. SAP forms, hospital EHR systems, that ancient JIRA instance. A computer-use agent unlocks all of it. Failure modes are new though: the model can't reliably click a small button in a busy UI, and hallucinated clicks are terrifying when the button says "Delete customer." Anthropic's release explicitly recommends running inside a sandboxed VM with human confirmation on any destructive action.

7.5 · Multi-agent frameworks (AutoGen, CrewAI, LangGraph)

2024's other trend: instead of one agent with all tools, spawn multiple specialized agents that message each other. AutoGen (Wu et al., Microsoft, arXiv:2308.08155), CrewAI, and LangGraph all provide the plumbing. The empirical evidence is mixed — Salesforce's Agent-as-a-Judge (arXiv:2410.10934) found multi-agent setups often increase error rates by 10–20% because each hop adds parsing noise. Use multi-agent when roles are genuinely different (planner vs coder vs reviewer), not just for the aesthetics.

7.6 · Evaluation — SWE-bench, GAIA, WebArena

"My agent works" needs a benchmark. The 2024–2025 standards:

• SWE-bench Verified (Jimenez et al., arXiv:2310.06770 + OpenAI's human-audited subset): fix real GitHub issues. Claude Sonnet 4 and GPT-5 sit around 65–75% pass rate as of mid-2025 — up from 2% in early 2024.
• GAIA (Mialon et al., arXiv:2311.12983): general-assistant multi-step reasoning with tools. Humans ~92%, top agents ~65%.
• WebArena / VisualWebArena (Zhou et al., arXiv:2307.13854): actual browser tasks in sandboxed sites. Humans ~78%, top agents ~35% — still wide open.

Run your agent on at least one of these before shipping. "Vibes-based" evaluation of agents is how people burn six figures on the wrong stack.

7.7 · Further reading

• ReAct: https://arxiv.org/abs/2210.03629 · Yao et al. 2022
• Reflexion: https://arxiv.org/abs/2303.11366 · Shinn et al. 2023
• Voyager: https://arxiv.org/abs/2305.16291 · Wang et al. 2023
• Toolformer: https://arxiv.org/abs/2302.04761 · Schick et al. 2023
• AutoGen: https://arxiv.org/abs/2308.08155 · Wu et al. 2023
• SWE-bench: https://arxiv.org/abs/2310.06770 · https://www.swebench.com/
• GAIA: https://arxiv.org/abs/2311.12983
• Anthropic Computer Use: https://www.anthropic.com/news/3-5-models-and-computer-use
• OpenAI Operator/CUA: https://openai.com/index/introducing-operator/
• Outlines (constrained decoding): https://arxiv.org/abs/2307.09702

8 · Try it yourself

Build the smallest useful agent this week:

1. Pick a real annoyance in your life that involves 3+ steps and a data-dependent branch. (Mine: "find papers cited by paper X that were also cited by paper Y, filter to last 12 months.")
2. Sketch the 2–4 tools it needs (search_arxiv, get_citations, intersect, filter_by_date).
3. Implement each tool as a plain Python function (~15 lines each).
4. Wire the 30-line run_agent loop from §2 with Claude or GPT native tool-calling.
5. Run it on 10 real instances. Log failures. Add one Reflexion-style retry.

Budget: one evening. Reward: you'll never again mistake "I chained LangChain runnables" for "I built an agent."

9 · Recap

An agent is a while loop around an LLM: generate → parse a tool call → run it → append the observation → repeat until the model says "done." That's it. ReAct proved this works with zero fine-tuning because "think then act" transcripts are all over the pretraining corpus. Every 2024–2025 advance (Reflexion, Voyager, computer-use, native tool-calling) is a variation on the observation channel — richer memory, growing skills, screenshots instead of strings. The failure modes are the same as any long-context system: hallucinated tools, infinite loops, context flooding. The mitigations are boring engineering: validate, cap, truncate, sandbox.

# same model, same tools, same task set - three loops
variants = {
  'raw'      : dict(on_error='append_traceback', max_steps=20, dedup=False),
  'terse'    : dict(on_error='append_short_hint', max_steps=20, dedup=False),
  'guarded'  : dict(on_error='append_short_hint', max_steps=8,  dedup=True),
}
for name, cfg in variants.items():
    r = run_suite(tasks, model=MODEL, **cfg)
    print(name, 'solved', r.solved, 'avg_steps', r.avg_steps,
          'avg_tokens', r.avg_tokens, 'loops_detected', r.loops)

🧠 Retention scaffold

One-line summary (write it in your own words): _______________________________

Spaced review: re-read §2 (the 30-line loop) + §4 (failure modes) in 24 hours. Revisit alongside a SWE-bench trajectory on day 7.

Next session (S076): an agent that can search is only as good as its retriever. We build RAG from scratch — chunker, index, reranker, generator — with no LangChain, so you know exactly what the frameworks are hiding.

Sticky note (keep on your desk): An agent is a while loop. Everything else — memory, tools, planners, multi-agent orchestration — is decoration on that loop.

Previous: ← DL S074 · Next: DL S076 → RAG from Scratch