Search Tech Journey

Find topics, journeys and posts

6-month learning plan7 / 130
back to blog
pythonbeginner 55m read

S007 · Functions — arguments, scope, closures

Functions are the atom of Python. This session covers the four kinds of arguments, the LEGB scope rule, closures, decorators as functions, and the interview-favourite tricks that trip up devs who thought they knew Python.

🧠SoftwareM01 · Python Foundations· Session 007 of 130 90 min

🎯 Write functions that read well, argue clearly, and don't have hidden state — plus read anyone else's function and predict what it will do.

Why this session exists

Functions look simple until they don't. Positional vs keyword vs *args vs **kwargs is a menu most devs pick from wrong. Scope + closures + late binding cause the same interview trick to fool the same people every year. And decorators — the thing that lets @app.route and @lru_cache exist — are literally just functions taking functions. This session installs the whole picture so you never fumble a signature or misread a closure again.

You will be able to
  • Design a function signature with the right mix of positional / keyword-only / defaults / *args / **kwargs on purpose.
  • Trace the LEGB name-resolution rule for any nested function without running the code.
  • Recognise a closure, know when Python captures by reference (surprise!), and fix the late-binding bug.
  • Write your own decorator from scratch, including one that takes arguments.
  • Predict what `f(*args, **kwargs)` will bind to, given a specific call site.

Prerequisites



(a) Intuition · 5 min

A function is a contract, not a code block
🌍 Real world

A restaurant menu item is a contract: ‘give me a burger and $12, get back a burger with fries.’ The kitchen (implementation) can change — same beef, different sauce — but the contract with the customer is stable.

A function is exactly that. The signature is the contract; the body is the kitchen. A good signature (clear names, sensible defaults, no surprise inputs) lets you swap the body a hundred times without breaking anyone.

💻 Code world

Formally: a function is a first-class object with a __call__. It has a signature (parameters + return type), a scope (its own namespace + a link to the enclosing one), and a body. You can pass it around, store it in dicts, return it from other functions.

That last property is what makes closures, decorators, callbacks, and higher-order functional programming possible in Python — the same feature.

The four argument shapes

Every parameter you'll ever write is one of these
  • Positional-only — before `/` in the signature. Callers cannot use the name. Rare, but useful for stable APIs.
  • Positional-or-keyword — the default. Can be passed either way.
  • Keyword-only — after `*` or `*args` in the signature. Callers MUST use the name. Great for boolean flags.
  • Var-args (`*args`, `**kwargs`) — arbitrary numbers of positional/keyword args. Use sparingly; explicit signatures beat implicit ones.

A quick history so you know the shape

  1. 1991
    Python 0.9.0 · def
    Functions from day one. Guido borrowed def/return from ABC, not C.
  2. 2001
    Decorators (PEP 318)
    Python 2.4 lands `@decorator` syntax — syntactic sugar for `f = decorator(f)`.
  3. 2006
    Keyword-only args (PEP 3102)
    Python 3.0 lets you enforce keyword-only args with `*`. Underused, superb for API design.
  4. 2015
    Type hints (PEP 484)
    Optional annotations arrive. Every serious codebase types their function signatures within 5 years.
  5. 2020
    Positional-only params (PEP 570)
    `/` in signatures. Lets libraries expose stable positional APIs without freezing the parameter names.

(b) Visual walkthrough · 15 min

The full function signature grammar

A full example:

def request(url, /, method="GET", *headers, timeout=30, **params):
    ...
 
# url is positional-only
# method is positional-or-keyword with a default
# *headers gathers any extra positional args into a tuple
# timeout is keyword-only (because it comes after *headers)
# **params gathers any extra keyword args into a dict

The LEGB scope resolution rule

Every name lookup walks L → E → G → B. Writes create a Local unless you say nonlocal (to write to an Enclosing) or global (to write to a Global). Memorise the order. Debugging weird scope bugs becomes trivial once you can point at the rung.

The five kinds of ‘function-like’ things

def

The default

  • Named
  • Any signature
  • Can have docstring
  • Debuggable
  • Use unless there's a specific reason not to
lambda

Anonymous, one-expression

  • Unnamed
  • One expression only (no statements)
  • Great for sort keys and small callbacks
  • Bad for anything more complex
  • PEP 8: don't assign to a name — use def
@decorator

A function that transforms a function

  • def deco(f): def wrap(*a, **k): return f(*a, **k); return wrap
  • Powers @app.route, @lru_cache, @staticmethod
  • Use functools.wraps to preserve __name__
  • Can be parameterised: @deco(arg)
functools.partial

Freeze some arguments

  • greet_bob = partial(greet, 'bob')
  • Cleaner than a lambda
  • Preserves the underlying function
  • Great for callbacks needing a specific signature
class __call__

A callable OBJECT

  • Class with __call__ method
  • Instances behave like functions
  • Can carry state between calls
  • How torch.nn.Module and Django views work

Closures — the ‘function remembers its home’ trick

1step 1
Define an inner function that uses an outer variable

def make_counter(): count = 0; def inc(): nonlocal count; count += 1; return count; return inc

2step 2
Return the inner function

The outer function returns `inc`. Normally `count` would die when make_counter returns.

3step 3
Python attaches `count` to `inc.__closure__`

The cell object keeps `count` alive as long as any reference to `inc` exists.

4step 4
Every call to inc() reads/writes the same cell

Two separate make_counter() calls give two independent counters — each has its own cell.

The mental model to hold


Common misconception
✗ What most people think

"A default argument is evaluated each time the function is called — so def f(items=[]) gives me a fresh empty list on every call."

✓ What is actually true

Default values are evaluated once, when the def statement executes, and stored on the function object in __defaults__. Every call that omits the argument gets the same object. A mutable default is shared state with a very long lifetime.

Why the myth is so sticky

Because it is completely invisible for immutable defaults, which is what you write 90% of the time. def f(n=0), def f(s=""), def f(flag=False) all behave identically under both models, because you can never mutate them — you can only rebind, which is local. The moment the default is a list, dict, or a call like datetime.now(), the two models diverge and the function starts remembering things across calls. It looks like a haunting: the second call returns data from the first, and there is nothing in the function body that stores anything.

Prove it to yourself

Watch the default itself change, and see where it lives:

def add(x, bucket=[]):
    bucket.append(x)
    return bucket

print(add(1))          # [1]
print(add(2))          # [1, 2]   <- not fresh
print(add.__defaults__)  # ([1, 2],)  <- it lives on the function

# the fix, and the reason 'None' is the idiom:
def add(x, bucket=None):
    bucket = [] if bucket is None else bucket
    bucket.append(x)
    return bucket
From first principles
Start with the question

Why does a closure capture the variable rather than its value — so that every function created in a loop ends up seeing the loop's final value? This is famously surprising; derive why it has to work this way.

  1. 1
    A nested function must be able to read names from its enclosing scope, otherwise you could not write a helper that uses its parent's arguments.
    forced by · lexical scoping is the whole point of nesting a function inside another
  2. 2
    The enclosing function's locals can change after the inner function is defined — the outer code keeps running, loops keep incrementing.
    forced by · definition is just another statement; execution continues past it
  3. 3
    So the language must choose: snapshot the value at definition time, or hold a reference to the variable's storage cell. Snapshotting would break the common case where an inner function is defined before the outer function finishes computing what it needs.
    forced by · a decorator or callback defined early must still see values assigned later
  4. 4
    Therefore Python stores a reference to the cell holding the variable, and resolves the name when the closure is called, not when it is created.
    forced by · late binding is the only rule that keeps mutually-referencing and recursively-defined nested functions working
  5. 5
    A loop reuses one cell for the loop variable across all iterations, so all N closures created in the loop share that single cell — and by the time any of them runs, the loop is over.
    forced by · the loop variable is one name in one scope, not a new name per iteration
⇒ Therefore

Therefore "all my lambdas return 9" is not a bug, it is late binding working exactly as specified. Verify with f.__closure__[0].cell_contents — you can see the shared cell.

And note what this predicts: the fix must be to create a new cell per iteration, which is exactly what a default argument does (lambda x, i=i: ...) because defaults are evaluated at definition time — the same mechanism that caused the mutable-default trap now saves you. It also predicts why functools.partial(f, i) works: it binds the value, not the name.

Mental modelA function is an object with a signature contract

Stop picturing def as "declaring a subroutine". def is an assignment statement: it builds a function object — carrying code, defaults, closure cells, annotations, a name — and binds it to a name. That object can be stored in a dict, passed as an argument, returned, and given attributes, exactly like a list can.

The signature is the object's public contract: which arguments are positional, which are keyword, which have defaults. Everything painful about Python functions — mutable defaults, late binding, decorators losing metadata, *args/**kwargs — follows from the object being built once and called many times.

  • Never use a mutable or time-dependent default. Use None as the sentinel and construct inside the body.
  • Arguments are passed by assignment: the parameter becomes another name for the caller's object. Rebinding is local; mutating is visible to the caller.
  • Use * in the signature to force keyword-only arguments for anything a reader couldn't decode at the call site — booleans especially. f(data, dropna=True) beats f(data, True).
  • A function should either return a value or perform an effect, not both. Returning None after mutating is Python's own convention (list.sort) precisely to make the choice legible.
🔔 Fires when you see

Fire this model the moment you see: a function that "remembers" its previous call · every closure in a loop giving the same answer · a decorated function whose __name__ is wrapper · a timestamp default that is always the import time · a caller's list changing unexpectedly.

The tradeoff

A function has grown to eight parameters. Do you keep the flat signature, take a config object, or split it into several functions?

Keep the flat signature (keyword-only)
+ you gain every input is explicit and discoverable via help() and IDE completion; type checkers verify each argument; no hidden coupling
− you pay call sites get long, adding a parameter touches every caller, and related parameters that must be set together are not visibly grouped
pick when the parameters are genuinely independent and fewer than roughly seven — the point at which a reader stops holding the signature in working memory
Config object / dataclass
+ you gain related settings travel together, get validated once at construction, are serialisable to JSON/YAML for reproducibility, and adding a field breaks nothing
− you pay indirection — you must open a second definition to know what is available; the object can arrive half-populated; it invites unrelated settings to accumulate
pick when the same parameter bundle is passed through three or more layers, or the settings need to be persisted, logged, or diffed — which is the normal case for training and pipeline configs
Split into several functions
+ you gain each function does one thing with a short signature, becomes independently testable, and boolean flags that switched behaviour disappear entirely
− you pay more names to know, and shared setup logic must be factored out or duplicated; callers now orchestrate what one call used to do
pick when a boolean parameter selects between two largely disjoint code paths — that flag is a function boundary wearing a disguise
What a senior engineer actually does

Look at the parameters before counting them. Flags that fork the body mean you have two functions. Parameters that always move together mean you have a missing type. Only genuinely independent knobs justify a wide signature — and those should be keyword-only so call sites stay readable.

In data and ML work the config-object answer wins more often than in general software, for a reason unrelated to code quality: a serialisable config is a record of the run. If the parameters exist only as positional arguments in a notebook cell, the experiment is not reproducible no matter how clean the function is.


(c) Hands-on · 25 min

Save as s007_functions.py, run, and PREDICT each output before it prints.

"""s007_functions.py — the full menu of Python function features."""
from __future__ import annotations
from functools import wraps, partial, lru_cache
import time
from typing import Callable, Any
 
# =========================
# Part 1 · Signature grammar
# =========================
print("--- Part 1 ---")
 
def request(url, /, method="GET", *headers, timeout=30, **params):
    return f"{method} {url} timeout={timeout} headers={headers} params={params}"
 
print(request("https://x", "POST", "H1", "H2", timeout=5, page=2, sort="new"))
# Try to call `request(url="x")` — it fails: url is positional-only.
 
# =========================
# Part 2 · Default arg trap (revisited from S005)
# =========================
print("\n--- Part 2 ---")
 
def bad(item, bin=[]):
    bin.append(item)
    return bin
 
def good(item, bin=None):
    bin = [] if bin is None else bin
    bin.append(item)
    return bin
 
print(bad("a"), bad("b"))     # shared default — PREDICT
print(good("a"), good("b"))   # fresh each call
 
# =========================
# Part 3 · Scope (LEGB) and nonlocal / global
# =========================
print("\n--- Part 3 ---")
 
x = "global"
 
def outer():
    x = "enclosing"
    def inner_read():
        # reads x from Enclosing (LEGB)
        return x
    def inner_write_local():
        # this creates a new LOCAL x — doesn't touch enclosing
        x = "local"
        return x
    def inner_write_nonlocal():
        nonlocal x
        x = "changed by nonlocal"
        return x
    print("read:", inner_read())
    print("write local:", inner_write_local(), "outer x:", x)
    print("write nonlocal:", inner_write_nonlocal(), "outer x:", x)
 
outer()
print("module x:", x)
 
# =========================
# Part 4 · The late-binding closure bug
# =========================
print("\n--- Part 4 ---")
 
# Bug — all three lambdas capture the SAME i
funcs = [lambda: i for i in range(3)]
print("bug:", [f() for f in funcs])
 
# Fix 1 — default argument snapshot
funcs = [lambda i=i: i for i in range(3)]
print("fix1:", [f() for f in funcs])
 
# Fix 2 — factory function
def make(i):
    def f():
        return i
    return f
funcs = [make(i) for i in range(3)]
print("fix2:", [f() for f in funcs])
 
# =========================
# Part 5 · Closures with state
# =========================
print("\n--- Part 5 ---")
 
def make_counter(start: int = 0) -> Callable[[], int]:
    count = start
    def inc() -> int:
        nonlocal count
        count += 1
        return count
    return inc
 
c1 = make_counter()
c2 = make_counter(100)
print(c1(), c1(), c1())
print(c2(), c2())
 
# =========================
# Part 6 · Decorator from scratch
# =========================
print("\n--- Part 6 ---")
 
def timed(func: Callable) -> Callable:
    @wraps(func)                # preserves __name__, __doc__
    def wrapper(*args: Any, **kwargs: Any) -> Any:
        t0 = time.perf_counter()
        result = func(*args, **kwargs)
        elapsed = (time.perf_counter() - t0) * 1000
        print(f"  {func.__name__} took {elapsed:.2f} ms")
        return result
    return wrapper
 
@timed
def slow_add(a: int, b: int) -> int:
    """Add two numbers slowly, for demo."""
    time.sleep(0.05)
    return a + b
 
print(slow_add(2, 3))
print("still named correctly:", slow_add.__name__)   # 'slow_add' thanks to @wraps
 
# Parameterised decorator
def repeat(n: int) -> Callable:
    def decorator(func: Callable) -> Callable:
        @wraps(func)
        def wrapper(*args: Any, **kwargs: Any) -> list:
            return [func(*args, **kwargs) for _ in range(n)]
        return wrapper
    return decorator
 
@repeat(3)
def greet(name: str) -> str:
    return f"hi {name}"
 
print(greet("ada"))
 
# =========================
# Part 7 · Higher-order plumbing
# =========================
print("\n--- Part 7 ---")
 
# functools.partial
add = lambda a, b: a + b
add5 = partial(add, 5)
print(add5(10))
 
# functools.lru_cache — a real-world decorator
@lru_cache(maxsize=128)
def fib(n: int) -> int:
    return n if n < 2 else fib(n - 1) + fib(n - 2)
 
print([fib(i) for i in range(10)])
print("cache stats:", fib.cache_info())
 
# =========================
# Part 8 · Callable classes
# =========================
print("\n--- Part 8 ---")
 
class Multiplier:
    """A callable that carries state."""
    def __init__(self, factor: int) -> None:
        self.factor = factor
    def __call__(self, x: int) -> int:
        return x * self.factor
 
times3 = Multiplier(3)
print(times3(10), times3(4))
print("is it callable?", callable(times3))

What each block teaches

Anatomy of the exercises

Part 1 · signature grammar
positional-only (before /), keyword-only (after *), *args, **kwargs. All four in one signature.
grammar
Part 2 · mutable defaults
The bug you saw in S005, in function form. The `= [] if x is None else x` idiom is muscle memory.
gotcha
Part 3 · LEGB in action
Read uses enclosing. Write creates local UNLESS you use nonlocal or global. The most common source of scope surprises.
scope
Part 4 · late binding
Closures capture VARIABLES not values. Default-arg trick or a factory function fixes it. Interview classic.
closure
Part 5 · closures with state
make_counter shows that each call to the outer produces a NEW enclosing scope, so counters are independent.
closure
Part 6 · decorators
A decorator is just `wrapped = deco(f)`. `functools.wraps` preserves the original name and docstring. Parameterised decorators are decorators that RETURN a decorator.
meta
Part 7 · higher-order stdlib
partial freezes args; lru_cache memoises. Both are decorators/HOFs you'll use weekly.
stdlib
Part 8 · callable classes
Any object with __call__ is a function. Useful when you need state + a function interface (e.g. torch.nn.Module).
advanced
Try itWrite your own `@retry(times=3, delay=0.5)` decorator

Write a decorator @retry(times=3, delay=0.5) that:

  1. Calls the wrapped function.
  2. If it raises, sleeps delay seconds and retries, up to times total attempts.
  3. After the last failure, re-raises the exception.
  4. On success, returns the result.

Test with a function that fails the first 2 calls and succeeds the third. Verify functools.wraps preserves the name.

💡 Hint · A parameterised decorator is a function that RETURNS a decorator. Three levels of `def`: outer takes params, middle takes the function, inner takes call-time *args/**kwargs.

(d) Production reality · 15 min

War story Django · reported repeatedly on the trackerViews randomly returning stale data
🔥 What broke

A view uses def get_data(request, filters={'active': True}). Under normal traffic, everything's fine. Under some request patterns, a helper inside get_data mutates the filters dict (adding user-specific keys). Second request: dict already has those keys — bug reproduces sometimes, not always.

🧯 The fix

Same fix as always: filters=None + filters = filters or {}. Bonus: enable ruff's B006 rule to catch mutable defaults at CI time.

Long-term: use types.MappingProxyType({}) for a truly-immutable default, or a frozenset() if the callers agree.

🎓 Lesson to steal
Mutable defaults in function signatures are a foot-gun that ships to production more than any other Python bug. Configure your linter to reject them.
War story A hiring manager · every yearInterview question that eliminates half of candidates
🔥 What broke

Interviewer writes on the whiteboard:

callbacks = []
for i in range(3):
    callbacks.append(lambda: i)
print([cb() for cb in callbacks])

‘What does this print, and why?’ Candidates who confidently say [0, 1, 2] fail. Candidates who spot the late-binding and can explain the fix get the offer.

🧯 The fix
Explain: closures capture the variable, not the value. All three lambdas share the same i. Fix with lambda i=i: i (default-arg snapshot) or a factory function.
🎓 Lesson to steal
This is the most-asked Python interview trick after ‘mutable defaults’. It's not a Python quirk — Java, JS, C# all had similar bugs (JS's `var` in a for-loop is the same class of bug). Know it, be able to explain it, be able to fix it two ways.
War story Flask + gunicorn setups · commonRandom 500 errors under load
🔥 What broke
A dev writes @app.before_request that mutates a module-level list. Under gunicorn with multiple workers, each worker has its own module-level state — which the dev didn't realise. Sessions get ‘forgotten’ intermittently as the load balancer routes to a fresh worker.
🧯 The fix

Any shared state that must be visible across workers goes in an external store (Redis, DB, memcached). Module-level Python state is per-process, and every serious deployment has multiple processes.

Alternative: gunicorn's --preload shares memory across workers, but only for READS — writes still fork on copy-on-write.

🎓 Lesson to steal
Function-scope state (closures) is per-call. Module-scope state is per-process. Cross-process state needs an out-of-process store. Get this wrong and your bugs are non-reproducible.

Where this shows up in the rest of the plan

Functions are the atom every future session builds on
S009 · Classes & objects
Methods are functions with `self`. Everything you learned today applies to methods.
S010 · Inheritance
Method resolution order (MRO) is LEGB's cousin for classes.
S013 · Testing
Fixtures in pytest are literally decorators + closures. You'll write dozens.
S057 · Flask/FastAPI
@app.route is a decorator. Middleware are decorators. Dependency injection is `Callable[Params, Result]`.
S068 · asyncio
async functions are functions too — with `await` and coroutines. Same signature grammar.
S099 · Code review
Reviewers judge you on function signatures. Get them right and half your PRs sail through.

(e) Recall + stretch · 10 min

Recall — click each to reveal · click to reveal
★ = stretch question

Explain-out-loud test

If you can't teach these three without notes, redo the session:

  1. What's a closure? (one sentence + one example)
  2. What's the difference between nonlocal and global? (one rule each)
  3. Write the shape of a decorator that takes arguments. (three levels of def)

What comes next

Hub: The 6-Month Learning Plan


Part of a 130-session evergreen learning series. Session structure: intuition → visual → hands-on → production war stories → recall. Duration: 90 minutes.