S047 · Spark — RDD, DataFrame, Jobs/Stages/Shuffles

Module M05: Data Engineering · Session 47 of 130 · Track: DE 🔥

What you'll be able to do after this session

• Explain Spark to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S046 — Batch vs Streaming — Mental Model & Use Cases

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · Apache Spark Architecture Explained — Driver, Executors, DAGs, Stages & Tasks — the whole architecture in one animated diagram.
• 🎥 Deep dive (30–60 min) · Advanced Apache Spark Training — Sameer Farooqui (Databricks) — the classic Databricks deep dive. Long, dense, worth every minute.
• 🎥 Hands-on demo (10–20 min) · Understand and Optimize Shuffle in Spark — real Spark UI, real shuffle spills, real fixes.
• 📖 Canonical article · Spark — RDD Programming Guide — the foundational RDD abstraction; every DataFrame op still compiles down to this.
• 📖 Book chapter / tutorial · Spark — SQL / DataFrame Programming Guide — the modern, primary API surface.
• 📖 Engineering blog · Databricks — Deep Dive: Catalyst & Tungsten — how Spark actually plans and optimizes your DataFrame code.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: The distributed compute engine that runs everything.

You have 10 terabytes of logs and one laptop. Loading them into pandas dies at line 3. Even a beefy 128 GB VM chokes. So you rent 100 modest machines and split the work — each machine reads 100 GB, computes its slice of the answer, and someone stitches the pieces back together. That "someone" plus "everyone" is Apache Spark: a distributed compute engine that turns a cluster of ordinary machines into one giant, coordinated calculator.

Spark's whole design is built around one insight: shipping the code to the data is cheaper than shipping the data to the code. Your Python or Scala program (the "driver") sends small blobs of computation to many "executor" processes on remote machines that already have the data. Data is split into "partitions" — chunks small enough to fit in memory. When possible, everything stays local ("narrow" transformations like filter, map); when data must move between machines to be grouped, joined, or sorted, that's a shuffle — the single most important word in Spark. Shuffles are where Spark jobs go to die.

The gotcha to carry forever: Spark is lazy, and every operation you write is a promise, not an action. Nothing runs until you call an action like .count(), .show(), .write.parquet(...). Before that, Spark stacks up your transformations into a DAG (a plan), optimizes them with Catalyst, and only then executes. This is why you can chain 50 DataFrame operations and it appears instant — the actual work happens on the last line. It's also why "my Spark job is slow" almost always means "one specific shuffle in the middle of a 20-step DAG is slow." Read the Spark UI. It tells you where the pain is.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

The Spark execution hierarchy, from the outside in:

• Application = one Spark session (one driver process).
• Job = one action. .count(), .write, .show() each create a job.
• Stage = a set of tasks that can run without a shuffle. A shuffle breaks a job into stages.
• Task = one unit of work on one partition, run on one executor core.

Narrow vs Wide transformations — the whole game:

Worked example — count events by country, from 1 TB of parquet:

Notice: the filter and partial group-by run locally on each executor (Stage 1, narrow). The shuffle boundary starts Stage 2. Getting a job to run in 1 stage instead of 3 often 10x-es performance — that's what "shuffle avoidance" means.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

Run local Spark and watch the DAG unfold.

pip install pyspark==3.5
python3 <<'PY'
from pyspark.sql import SparkSession, functions as F
 
spark = (SparkSession.builder
         .appName("s047-lab")
         .master("local[4]")               # 4 local worker cores
         .config("spark.sql.shuffle.partitions", "8")  # default is 200 — too many for a demo
         .getOrCreate())
spark.sparkContext.setLogLevel("WARN")
 
# 1. Generate 5M synthetic events
df = (spark.range(0, 5_000_000)
      .withColumn("country", F.element_at(F.array(F.lit("IN"),F.lit("US"),F.lit("DE"),F.lit("JP"),F.lit("BR")),
                                          (F.col("id") % 5 + 1).cast("int")))
      .withColumn("amount",  (F.rand() * 1000).cast("decimal(10,2)")))
 
# 2. LAZY — no work happens yet. Print the plan the optimizer built.
agg = df.filter(F.col("amount") > 100).groupBy("country").agg(F.sum("amount").alias("total"))
agg.explain(mode="formatted")
 
# 3. ACTION — now Spark actually runs the DAG
print(agg.orderBy(F.col("total").desc()).show())
 
# 4. Force a shuffle you don't want (a bad repartition)
df.repartition(200, "country").groupBy("country").count().show()
# Open http://localhost:4040 in another tab to see the Jobs / Stages tabs while this runs
 
# 5. Broadcast join — no shuffle for the small side
small = spark.createDataFrame([("IN","India"),("US","USA")], ["country","full"])
df.join(F.broadcast(small), "country").select("country","full","amount").show(3)
 
input("Press Enter to shut down (leave Spark UI open at :4040)... ")
spark.stop()
PY

Observe (checklist):

1. explain() shows the physical plan with Exchange hashpartitioning(country, 8) — that Exchange is the shuffle. See it before you run it.
2. Open http://localhost:4040 while a job runs — the Jobs → Stages → Tasks tabs show exactly where time is spent.
3. In step 4, repartition(200, ...) creates 200 tiny tasks — the UI shows tons of scheduler overhead. Fewer, bigger tasks are almost always faster.
4. In step 5, broadcast(small) avoids a shuffle by sending the small table to every executor. Spark auto-broadcasts tables under ~10 MB; you can hint it explicitly.
5. Watch the "shuffle read/write" columns per stage — this is your #1 debugging signal.

Try this modification: create a heavily-skewed column (e.g., 99% of rows have country='US') and groupBy it. Watch one task take 10x longer than the others in the Spark UI. Then try spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true") and rerun. This is Spark 3's Adaptive Query Execution rescuing you.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

Every senior data engineer has one Spark war story that ends with "we increased spark.sql.shuffle.partitions from 200 to 2000 and the job went from 6 hours to 20 minutes." The reverse also happens. The default of 200 is optimized for nothing — for a 500 GB shuffle you probably want 2,000; for a 5 GB shuffle you probably want 50. Rule of thumb: aim for each shuffle partition to be ~100–200 MB post-shuffle. AQE (Adaptive Query Execution, on by default in Spark 3.2+) auto-coalesces this for you, but you still tune the initial count when AQE isn't enough.

The #1 root cause of slow Spark jobs in production is data skew — one key has 100x more rows than every other key. That one key becomes one massive task, and while 199 other tasks finish in seconds, that task takes hours. Symptoms: Spark UI shows one task with duration and shuffle read wildly higher than the rest. Fixes: enable AQE skew join, salt the key (add a random suffix, shuffle, then de-salt), or split the hot key out and process it separately.

Second-most-common: out-of-memory executor kills. Container killed by YARN for exceeding memory limits. Root causes: too-large a broadcast; a collect() that pulls 20 GB to the driver; a groupBy on a super-high-cardinality column. Never collect() unless you're 100% sure the result fits. Use .show(), .take(n), or write to storage.

Modern practice: prefer DataFrames over RDDs, always. The Catalyst optimizer rewrites your DataFrame code (pushes filters into scans, prunes columns, picks join strategies) — RDD code bypasses all of that. Also, write Parquet, not CSV — columnar formats let Spark push filters and column-pruning down into the file reader, often 20-100x faster. And on cloud object stores (S3, ADLS, GCS): watch out for the "small file problem" — 1 million 100 KB files are 100x slower to list and read than 100 files of 1 GB each. Compact regularly (this is one big reason Delta Lake / Iceberg exist).

At Microsoft on ODSP-scale data (300B+ transactions/day), the operational rule was: profile with the Spark UI first, tune shuffle partitions, watch for skew, avoid collect, and only then reach for cluster resizing. That order of operations solves 95% of "Spark is slow" tickets.

(e) Quiz + exercise · 10 min

1. What is the difference between a Job, a Stage, and a Task in Spark?
2. Define "narrow" vs "wide" transformation and give one example of each.
3. Why is Spark's execution model called "lazy," and what triggers actual computation?
4. What is a shuffle, and why is it usually the most expensive part of a Spark job?
5. What is a broadcast join and when does it dramatically outperform a regular join?

Stretch (connects to S043 — Query Planning): Spark's Catalyst optimizer plays the same role as a database query planner. Compare and contrast: what does Catalyst do that a Postgres planner also does, and what does it do that Postgres cannot (or does not need to)?

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Spark? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S048): Kafka — Topics, Partitions, Consumer Groups
• Previous (S046): Batch vs Streaming — Mental Model & Use Cases
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.