LLM Agents — Function Calling, Tool Use, Orchestration

Why this session matters

Part of a 24-topic learning series on engineering, ML, and LLM systems. Each session is a 90-minute deep-dive on one topic — designed so anyone can pick it up cold. Every two topics are followed by a revision session with recall prompts and hands-on drills.

Why this session matters

It builds on the rhythm of one focused topic, paced so you have time to actually absorb it rather than rush.

Agenda

• From chat to tools — function-calling primitives in modern APIs
• The agent loop — plan, act, observe, repeat
• Designing tools — schemas, idempotency, error surfaces
• Stopping the loop — budgets, max-iterations, escape valves
• Patterns — ReAct, Plan-and-Execute, Multi-Agent, Tool Routing

Pre-read (skim before the session)

• OpenAI — Function calling guide
• Anthropic — Claude tool use
• ReAct: Synergizing Reasoning and Acting (Yao et al., 2022)
• Anthropic — Building effective agents (2024)

Deep dive

1. From completion to call

The base LLM emits text. Function calling = teach it to emit structured JSON that names a tool and its arguments, instead of (or in addition to) prose.

{
  "tool_calls": [{
    "id": "call_a1",
    "type": "function",
    "function": {
      "name": "get_weather",
      "arguments": "{\"city\": \"Hyderabad\"}"
    }
  }]
}

You parse, execute, feed the result back as a "tool" message; the model picks up where it left off. This loop is the substrate of every "agent".

2. Anatomy of an agent step

loop:
  messages: [system, user, ... assistant tool_call, tool result, ...]
  response = llm.complete(messages, tools=TOOLS)
  if response.tool_calls:
     for call in response.tool_calls:
         result = dispatch(call)
         messages.append({role: "tool", call_id: call.id, content: result})
     continue
  else:
     return response.content   # final answer

Three things to manage carefully:

1. The state (messages list — grows; trim or summarise).
2. The tool dispatch (sync or async; sandbox if untrusted).
3. The stopping condition (max iterations, budget, idempotency check).

3. Tool schema design

Treat each tool as a small API:

{
  "type": "function",
  "function": {
    "name": "search_orders",
    "description": "Search orders by customer email or order id. Returns up to 20 matching orders.",
    "parameters": {
      "type": "object",
      "properties": {
        "email":    {"type": "string", "description": "Customer email exact match"},
        "order_id": {"type": "string", "description": "Order id, format ORD-12345"}
      },
      "required": []
    }
  }
}

Rules of thumb:

• Names verb-like and unambiguous — search_orders not orders.
• Descriptions matter more than you think. They're the model's only signal. Write them as if for a junior dev.
• Examples in the description when ambiguous — "format ORD-12345" beats explaining in prose.
• Few wide tools beat many narrow ones. query_database(sql) is a bad single tool (unsafe + too open). search_orders, get_customer, cancel_order is right.

4. Idempotency and safety

Agents will retry. They will hallucinate. They will call a tool twice with the same arguments after a partial failure. Make every state-changing tool idempotent by accepting a client-supplied request id:

{
  "name": "refund_payment",
  "parameters": {
    "payment_id": "pi_abc",
    "amount": 1000,
    "idempotency_key": "agent_run_xyz_step_3"
  }
}

For destructive actions, require confirmation: tool returns a confirmation token, model asks the user, calls again with token. Or wrap in a human-in-the-loop approval queue.

5. The stopping problem

Naive agent: while-true with no exit. Bad. Always cap:

MAX_ITER = 10
MAX_TOOLS_PER_TURN = 5
TOKEN_BUDGET = 50_000

for i in range(MAX_ITER):
    resp = llm.complete(messages, tools=TOOLS, max_tokens=...)
    if usage.total_tokens > TOKEN_BUDGET: bail("budget")
    if not resp.tool_calls: return resp.content
    if len(resp.tool_calls) > MAX_TOOLS_PER_TURN: bail("too many parallel calls")
    ... dispatch ...

bail("max iterations")

Loop without budget is a billing pager incident waiting to happen.

6. ReAct pattern

Yao et al. 2022 — interleave reasoning and acting. The model produces a "Thought:" before each tool call.

Thought: The user wants their last 3 orders. I should search by email.
Action: search_orders(email="dinesh@example.com")
Observation: [3 orders returned]
Thought: I have the orders. Format them clearly.
Final Answer: Here are your last 3 orders: ...

Modern function-calling APIs implicitly do this. ReAct is more useful when:

• Tools are expensive (model thinks before calling).
• You want auditable reasoning trails.

7. Plan-and-Execute

Two-stage agent:

1. Planner: produces a step list (no tool calls yet).
2. Executor: walks the steps, calling tools.

Cleaner for multi-step tasks (research, build me X). Cost: extra LLM call upfront. Lose flexibility if mid-plan you learn something new — usually mitigated by allowing re-plan.

8. Multi-agent — when (rarely)

The hype: spin up 5 agents that "collaborate". Reality: each new agent multiplies the cost and the chance of cascade failures. Most workflows are better as one well-tooled agent.

Multi-agent earns its keep when:

• Specialisation (researcher + writer + critic with very different prompts).
• Privilege separation (one agent can write code, another only review).
• Parallel exploration (5 agents try different approaches, pick best).

Default: one agent. Reach for multi-agent when forced.

9. Tool routing — the LLM is the router

When you have 100 tools, the system prompt + schemas explode and accuracy drops. Two patterns:

• Hierarchical menus — model first picks a category (orders, billing, accounts); category tool returns sub-tools.
• Retrieval-augmented tools — embed tool descriptions; at each step retrieve top-k relevant tools and only show those to the model.

Both keep the prompt small.

10. Production patterns from real agents

• Persist tool results. When a step fails mid-loop and you retry, replay observations.
• Audit log everything. Every prompt, every response, every tool call, every result. Token by token.
• Streaming — stream the model's text + tool calls; show progress to users. Latency feels half.
• Cancellation — user closes the chat; cancel any in-flight tool. Hand back partial state.
• Cost meter — every step prints tokens + $. Production-critical. Without it, agents silently 10x in cost overnight.

11. Evaluations for agents

Agents are notoriously hard to eval (covered more in S25). Quick wins:

• End-to-end task success — does the agent accomplish a labelled task?
• Tool-call F1 — did it call the right tool with right args, ignoring extra calls?
• Step efficiency — did it solve in N steps or 3N?
• LLM-as-judge for trace quality ("was this reasoning sensible?").

12. Frameworks landscape

My bias: start with a 50-line custom loop. Reach for a framework only when state/graph complexity demands it.

Reading material

Books:

• AI Engineering — Chip Huyen, 2024 (chs. on agents, tool use, evaluation — the best current overview)
• Building LLMs for Production — Louis-François Bouchard & Louïs Peters (the chapter on agentic frameworks)
• Designing Machine Learning Systems — Chip Huyen (the systems thinking pairs perfectly with agents)

Papers:

• ReAct: Synergizing Reasoning and Acting in Language Models — Yao et al. 2022 — the thought→action→observation loop every agent now uses.
• Toolformer: Language Models Can Teach Themselves to Use Tools — Schick et al. 2023 — self-supervised tool use.
• Reflexion: Language Agents with Verbal Reinforcement Learning — Shinn et al. 2023 — self-critique loop for agents.
• Voyager: An Open-Ended Embodied Agent with LLMs — Wang et al. 2023 — the GPT-4 Minecraft agent with skill library.
• SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering — Yang et al. 2024 (Princeton) — the agent that solves real GitHub issues.

Official docs:

• OpenAI — Function calling guide — the JSON-schema tool-call API.
• Anthropic — Tool use docs — Claude's tool-calling reference.
• Anthropic — Building effective agents — design patterns (workflows vs agents) from Anthropic.
• LangGraph — docs — stateful, graph-based agent framework.
• Model Context Protocol (MCP) — Anthropic spec — the emerging standard for tool servers.

Blog posts:

• The Bitter Lesson of LLM Agents — Simon Willison — the practitioner's pithy summary of Anthropic's post.
• Lilian Weng — LLM Powered Autonomous Agents — the canonical agent survey post; planning, memory, tools.
• OpenAI — A practical guide to building agents — OpenAI's own playbook (2025).

In-depth research material

• SWE-agent — github.com/princeton-nlp/SWE-agent — ~17k ★, the Princeton agent that solves SWE-bench issues.
• Open Interpreter — github.com/OpenInterpreter/open-interpreter — ~59k ★, local code-execution agent.
• smolagents (HF) — github.com/huggingface/smolagents — ~22k ★, minimalist code-action agents from HF.
• crewAI — github.com/crewAIInc/crewAI — ~37k ★, multi-agent orchestration with roles.
• LangGraph — github.com/langchain-ai/langgraph — ~19k ★, the graph-state agent runtime.
• Tau-bench — Sierra (Bret Taylor's company) — the canonical real-world agent benchmark.
• GAIA: A Benchmark for General AI Assistants — Mialon et al. 2023 — the agentic benchmark Meta+HF use.
• Computer Use API — Anthropic blog — screenshots + mouse/keyboard as a tool.
• Hamel Husain — Your AI Product Needs Evals — why agent evals are the hardest part.
• Cognition — Don't Build Multi-Agents — the influential counter-take from the Devin team.

Videos

• How I Use "AI Agents" (Real World Examples) — Andrej Karpathy — Andrej Karpathy · 39 min — the practitioner's view from a co-author of GPT.
• Agentic Design Patterns — Andrew Ng keynote — Andrew Ng · 30 min — reflection, tool use, planning, multi-agent: the four-pattern taxonomy.
• Building Effective Agents — Anthropic (Erik Schluntz) — 38 min — walk-through of the Anthropic post by one of its authors.
• State of GPT — Andrej Karpathy (Microsoft Build) — 42 min — the agent-loop framing in his GPT tutorial; still the clearest explainer.
• Function Calling and ReAct in Practice — Jerry Liu (LlamaIndex) — 49 min — from the founder of LlamaIndex; hands-on examples.

LeetCode — Basic Calculator Ii

• Link: https://leetcode.com/problems/basic-calculator-ii/
• Difficulty: Medium
• Why this problem: Stack-based eval respecting precedence — same logic an agent uses to evaluate tool chains.
• Time-box: 30 minutes. Look up the editorial only after.

Assignment / Deliverables

Give yourself a clean 2-hour window and complete all of these before moving on:

1. Read the deep-dive above end-to-end — no skimming. Take notes in your own words.
2. Solve the LeetCode problem below without help first. Only look at the hint after 15 focused minutes; only look at editorial after 30. Log your time.
3. Reproduce one code snippet locally. Pick the snippet that felt least obvious and get it running in a scratch file / notebook.
4. Draw the core diagram from memory. Paper, whiteboard, or tldraw — doesn't matter. If you can't, re-read section 2 and try again.
5. Write a 3-line takeaway in your prep journal: what surprised you, what you still don't understand, what you'd read next.
6. Skim one item from the Reading material section. Bookmark the rest for the weekend.
7. Commit any code + notes to your prep repo with message session-NN: <one-line summary>.

Stretch (optional, +30 min): explain today's topic to a rubber duck / a friend / a voice note. If you can't teach it in 5 minutes, you don't own it yet — flag it and revisit next weekend.

Post-session checklist

By the end of this session you should be able to:

• Implement a tool-use loop in <50 lines with iteration cap and token budget.
• Design 3 tool schemas with good descriptions; explain why each is right.
• Make every state-changing tool idempotent.
• Explain ReAct vs Plan-and-Execute and when to pick each.
• List 4 stopping conditions every agent loop must enforce.
• Solve basic-calculator-ii — stack-based eval, same shape as evaluating a tool chain with precedence.

Generated from sessions_data.py + content_part*.py. To edit a video / leetcode / title, edit the data file and re-run write_sessions.py.