DL S052 · Renting Your First GPU — Runpod / Modal / Vast.ai Walkthrough

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 52 / 80 · Module M09 · ~2 hours

The story we're starting with

Here is the moment your DL journey actually gets expensive: the first time you click "Deploy A100" on someone's website. There is a real credit card in your wallet, a real spot bidder somewhere waiting to pull the plug, and a real chance you forget the machine is running for six days and wake up to a $400 bill. This session's job is to make that transition safe.

You have three good options today for on-demand LLM training GPUs, in increasing order of hand-holding: Vast.ai (cheapest, sketchiest — think eBay for someone's home gaming rig), Runpod (mid-tier — proper data center hardware, decent UI, spot pricing), and Modal (most abstracted — Python-native, serverless GPUs, no SSH). All three will happily rent you an A100 for $1–2/hr. AWS/GCP/Azure will charge you$3–4/hr for the same thing plus enterprise-grade billing complexity you don't need.

We'll walk through Runpod as the default choice, then note where Modal differs. You'll learn: how to pick a template, how to sync a codebase in under 60 seconds, how to run a training job that resumes cleanly from a spot-interruption, and — most importantly — the exact discipline that means you never leave a GPU running after you're done.

Prerequisites

• S051 (DDP/FSDP) — you know what "multi-GPU" would even mean.
• S017 (training loop, logging) — you have wandb hooked up.

1 · Which GPU do you actually want?

For our 100M-param M09 run, target ~2B tokens. From S047: C ≈ 1.2 × 10¹⁸ FLOPs.

Two surprises here. First: consumer 3090/4090 are the cheapest per FLOP if your model fits in 24 GB. Second: H100 is often the cheapest total because it finishes so fast that even at $2.50/hr the hourly premium is smaller than the wall-clock savings.

For our M09 100M run, I recommend 4090 on Vast.ai for exploration (cheap enough to burn on debugging) and A100 40GB on Runpod for the final run (reliable, spot-safe, easy checkpointing).

2 · Runpod — the walkthrough

2.1 Sign up and load credit

• Go to runpod.io → Sign up.
• Add $20 to your account. That's ~15 A100-hours of runway; plenty for M09 with room for mistakes.
• Do not enable auto-refill. Set a hard cap.

2.2 Pick a template

Runpod's "Pod" is a container with a GPU attached. Under Pods → + Deploy → GPU Cloud you pick:

• GPU type: A100 40GB PCIe (or SXM if you want intra-node NVLink; usually not needed for one GPU).
• Region: cheapest available. Look for CA-OR-1 or EU-SE-1; often $0.30/hr less than US-East.
• Template: runpod/pytorch:2.4.0-py3.11-cuda12.4.1-devel-ubuntu22.04. Preinstalled CUDA + PyTorch + Python — saves 20 min setup.
• Volume: attach a 50 GB Network Volume (~$5/month). This survives pod termination — put your dataset, checkpoints, HF cache here.

2.3 Choose spot or on-demand

• On-demand: ~$1.50/hr for A100 40GB. Won't be interrupted.
• Spot (Interruptible): ~$0.65/hr — less than half. May be killed with 5 seconds notice.

For our run: spot. Because our training loop checkpoints every 500 steps (see S055) and can resume, spot interruption costs at most 10 minutes of work.

2.4 Click Deploy

30–90 seconds. When status shows "Running", you get an SSH command like:

ssh root@148.72.14.208 -p 15234 -i ~/.ssh/id_ed25519

Runpod injects your saved public key automatically. First-time users: paste your public key into Runpod's Settings → SSH Keys before deploying.

Log in. Run nvidia-smi. You should see one A100 in the output. Congrats — you're now paying $1.10/hour for a GPU. Every minute counts.

3 · The 60-second code sync

Two flavors, pick one:

3.1 Via git (recommended for clean pushes)

# On the pod
git clone https://github.com/yourname/dl-pretrain.git
cd dl-pretrain
pip install -r requirements.txt

3.2 Via rsync (best for iterative dev)

# On your laptop
rsync -avz --exclude '.git' --exclude '__pycache__' \
  -e "ssh -p 15234 -i ~/.ssh/id_ed25519" \
  ./dl-pretrain/ root@148.72.14.208:/workspace/dl-pretrain/

Every code change: rsync again. ~2 seconds. Beats git-push-git-pull cycles.

3.3 The persistent-volume trick for datasets

HuggingFace datasets by default cache to ~/.cache/huggingface — but that's on the ephemeral pod disk. Redirect to the persistent volume:

export HF_HOME=/workspace/hf-cache
export TRANSFORMERS_CACHE=/workspace/hf-cache

Put those in ~/.bashrc on the pod. Now if you kill and re-spin the pod, your (huge) tokenized dataset survives. Saves you from re-downloading a 20 GB corpus every launch.

4 · Modal — the anti-pod paradigm

Modal is different. There's no SSH, no pod. You write a Python function, decorate it, and Modal spins up a GPU only for the duration of that function.

import modal
 
app = modal.App("pretrain-100m")
image = modal.Image.debian_slim().pip_install("torch", "wandb", "transformers")
 
@app.function(image=image, gpu="A100", timeout=3600 * 8, volumes={"/data": modal.Volume.from_name("pretrain-data")})
def train():
    import torch
    # ... your training loop
    torch.save(model.state_dict(), "/data/final.pt")
 
@app.local_entrypoint()
def main():
    train.remote()

Then:

modal run train.py

Modal handles image build, GPU acquisition, code sync (from your local dir), and — critically — automatically shuts down the GPU when your function returns. You cannot leave it running by accident. Billing is per-second.

Pricing: A100 40GB at $2.65/hr, which is 2× Runpod spot. You pay a premium for the guaranteed auto-shutdown and the zero-ops experience. For long-lived training this eats up. For short experiments (~1 hour), the premium is worth the peace of mind.

Recommendation: use Modal for one-shot experiments and short fine-tunes; use Runpod for anything longer than 4 hours.

5 · Wandb — see your loss from your phone

Sign up at wandb.ai, get an API key. On the pod:

pip install wandb
wandb login    # paste key

In your training loop:

import wandb
wandb.init(project="pretrain-100m", name=f"run-{time.strftime('%Y%m%d-%H%M')}")
# ... at each step:
wandb.log({"loss": loss.item(), "lr": scheduler.get_last_lr()[0], "step": step})

Now you can wandb.ai/yourname/pretrain-100m in a browser and watch loss live. Also open on your phone — I check runs from bed. When the loss curve does something weird you get a signal within seconds.

6 · Cost tracking discipline

The one-line habit that will save you money. Every time you SSH into a pod, run:

echo "STARTED $(date) — REMEMBER TO KILL"

Or, better, set up an autostop:

# Auto-shutdown after 8 hours
(sleep 28800 && sudo poweroff) &

If your training completes in 3 hours, the pod dies after 8 no matter what.

The Modal equivalent: @app.function(timeout=3600 * 8). Same idea, enforced by the platform.

6.1 The "am I really done?" checklist before terminate

Before you click Stop / Terminate:

1. Checkpoint saved? ls -lh /workspace/checkpoints/*.pt — does the timestamp match now-ish?
2. wandb finalized? Check the dashboard shows "finished" status — otherwise last log entries may be lost.
3. Copied off the pod? Run once: rsync -avz -e "ssh -p 15234" root@ip:/workspace/checkpoints/ ./local-checkpoints/. If it's on your laptop, it's safe.
4. Deleted the pod, not just stopped? Runpod bills for storage on stopped-but-not-terminated pods. If you're really done, click Terminate.

I have a laminated version of this taped near my desk. Learn from my $180 mistake.

7 · A real one-shot training kick-off

Here's the actual command sequence for a 100M pretraining launch:

# ------ On my laptop ------
# 1. Sync code
rsync -avz -e "ssh -p 15234 -i ~/.ssh/id_ed25519" \
  ./dl-pretrain/ root@148.72.14.208:/workspace/dl-pretrain/
 
# ------ On the pod ------
# 2. Activate env, verify GPU
source ~/.bashrc
cd /workspace/dl-pretrain
nvidia-smi | head -20
 
# 3. Pre-tokenize once (skip if data already sharded)
python tokenize_corpus.py --input /workspace/data/raw --output /workspace/data/shards
 
# 4. Launch training in a tmux so SSH disconnects don't kill it
tmux new -s train
python train.py \
  --config configs/100m.yaml \
  --data /workspace/data/shards \
  --checkpoint_dir /workspace/checkpoints \
  --resume_from /workspace/checkpoints/latest.pt 2>/dev/null || true
# Ctrl-b d to detach
 
# 5. Follow along
tail -f /workspace/dl-pretrain/train.log
# Or just open wandb.ai in your browser

Detach from tmux, close the SSH connection, walk away. Come back in 3 hours and pull the checkpoint.

8 · Spot-interruption survival

Spot pods can die with 5 seconds of notice on Runpod. Two defenses:

1. Checkpoint often. Every 500 steps for a ~5M-token-per-step training run = every ~2 minutes. Worst-case loss on interrupt: 2 minutes of compute.
2. Auto-resume launcher. Wrap your training script in a shell loop:

while true; do
    python train.py --resume_from /workspace/checkpoints/latest.pt || true
    # If exit code is 0, training completed. Break out.
    if [ -f /workspace/checkpoints/DONE ]; then
        break
    fi
    echo "Restarting after interruption at $(date)"
    sleep 5
done

Your train.py writes DONE when it hits the final step. Otherwise the loop keeps trying. Combine with automatic pod restart on Runpod (setting: "restart on spot reclaim → yes") and your run is functionally uninterruptible.

9 · Real dollar totals for the M09 pipeline

If everything goes right:

Add 30% for debugging and re-runs → **$15 for the full 100M capstone**. Cheaper than dinner.

10 · War stories

11 · 2025 GPU rental landscape — real prices, real receipts

GPU pricing moves monthly. These are the numbers as of mid-2025 for the providers I've personally used (or my colleagues have) in the last 90 days. Always confirm at provider websites before you commit — H100 spot prices in particular have swung 40% in a quarter as supply came online.

11.1 · The provider matrix (mid-2025)

Sanity check across the row: for a 1×A100-80GB running 24 hours, RunPod spot = $19, Lambda on-demand =$26, AWS p4d spot = $38, AWS p4d on-demand =$98. Choose the row that matches your risk tolerance and your appetite for babysitting.

11.2 · Cost calibration for real 2025 pretrains

Let me anchor these numbers to something concrete. Here are back-of-envelope costs for training named 2024–2025 open models, computed as active_params × tokens × 6 ÷ (MFU × GPU_FLOPs) ÷ 3600 × $/GPU-hr.

The "your run" line is the point of this whole module. You can pretrain a functional 100M-param language model for the cost of a coffee. Frontier labs spend 6 orders of magnitude more; you learn 80% of the same lessons.

11.3 · Modal for the impatient — the serverless training pattern

Modal (modal.com) is the outlier in the table — it's not "rent a pod for N hours," it's "decorate a Python function and Modal spins up a GPU only while it runs." For M09-scale runs (2–4 hours), this can be cheaper and less error-prone than a pod because:

• You literally cannot forget to shut it down. Function returns → GPU released.
• Persistent volumes (modal.Volume) work identically to Runpod's network volumes.
• Wandb, hf-cache, checkpoints all mount naturally.
• Free tier: $30/month, enough for ~15 hours of A100 or ~10 hours of H100.

A minimal Modal training entrypoint looks like:

import modal
app = modal.App("pretrain-100m")
img = modal.Image.debian_slim().pip_install("torch==2.5.1", "transformers", "wandb", "datasets")
vol = modal.Volume.from_name("pretrain-store", create_if_missing=True)
 
@app.function(gpu="A100-40GB", image=img, volumes={"/vol": vol},
              timeout=4*60*60, secrets=[modal.Secret.from_name("wandb")])
def train(steps: int = 6000):
    import subprocess
    subprocess.run(["python", "/vol/train.py", "--steps", str(steps), "--out", "/vol/ckpts"], check=True)
 
@app.local_entrypoint()
def main():
    train.remote(steps=6000)

Run with modal run train.py. When the function exits, the GPU is gone. If your laptop crashes, the job keeps running on Modal (view logs at modal.com dashboard).

11.4 · TogetherCompute for the "I want someone else to babysit" case

Together AI (together.ai) offers a genuinely different model: you upload a config, they run the pretrain on their cluster, you get a checkpoint back. Their pricing for dedicated H100 clusters is ~$2.10–$2.40/GPU-hr with real InfiniBand and their in-house monitoring. They also open-sourced their RedPajama dataset (their 30T-token replication of the Llama pretraining mix) and their training framework, so nothing is locked in. For a solo 100M run it's overkill; if you ever plan to scale to 7B, having a relationship with a Together account manager saves weeks.

Further reading (11.x)

• Runpod pricing — runpod.io/pricing
• Vast.ai console — vast.ai (search by DL perf + reliability score)
• Modal LLM training example — modal.com/docs/examples/llm-finetuning
• Lambda Cloud pricing — lambdalabs.com/service/gpu-cloud
• Karpathy's llm.c cost report (GPT-2 for $600) — github.com/karpathy/llm.c/discussions/677
• DeepSeek-V3 training cost breakdown — tech report §5.4

12 · 🧠 Retention scaffold

One-line summary (write it in your own words): _______________________________

Spaced review: re-read §1 (which GPU) and §11 (2025 prices) in 24 hours. Revisit the full session on day 7.

Next session (S053): you pull the trigger — a real 100M-param pretraining run, launched, monitored, finished, with actual loss curves and actual numbers.

Sticky note (keep on your desk): "Network Volume + spot + auto-resume + wandb dashboard + phone timer at 2× ETA = safe rental."

Legacy recall drills (kept for spaced-review continuity)

Stretch

Pick a real published model (Mistral 7B, Phi-3-mini, Llama-3-8B) and estimate: what would it cost to reproduce their pretraining from scratch on Runpod spot, at MFU 40%? Break down (a) GPU-hours from Chinchilla and (b) $ at current spot prices. Then double it (real MFU is usually 30-45% and debug reruns add ~30%).

In your own words

"The two habits that keep GPU rental affordable are ____ and ____."

Spaced-review pointer

• S047 (Chinchilla) — compute → cost. All your budgeting comes from that formula.
• S051 (DDP/FSDP) — whether you need one GPU or eight. For M09, one is enough.

Next-session teaser

You have a rented A100 and a training script. Session 053 pulls the trigger — a full 100M-param pretraining, launched, monitored, and finished, with real loss curves and real numbers. This is the payoff for Sessions 1–52.

Bring back tomorrow

• Number: A100 40GB spot ≈ $1.10/hr on Runpod.
• Command: (sleep 28800 && poweroff) & right after SSH.
• Checklist: the 4-step "am I really done" before terminate.

Previous: ← DL S051 · Next: DL S053 →