Renting Your First GPU — Runpod / Modal / Vast.ai Walkthrough

The first rental is an operations experiment

The moment a provider allocates a GPU, an independent billing clock may start. Closing SSH, stopping Python and stopping a bill are three different events. The goal is not “deploy in five minutes”: it is to select a resource from a measured workload, transfer only permitted material, launch reproducibly, monitor progress, survive interruption and prove teardown without losing the only checkpoint.

Prerequisites are a working local training loop, a small memory/throughput pilot, SSH public-key concepts and the DDP/FSDP state inventory. Every cloud command and provider procedure here is a reference only, not executed in this review. No account was funded, GPU rented, model weights or datasets downloaded, paid API called or remote credentials entered. Provider documentation is a snapshot accessed 2026-10-08, not a frozen promise about future interfaces or prices. The useful rental workflow remains taught even though the service was not reproduced.

Choose the GPU from the whole memory inventory

A100 40 GB and 80 GB are different capacity choices; H100 also comes in variants whose memory, interconnect and arithmetic specifications differ. RTX 4090 has 24 GB of GDDR6X. Those labels do not determine whether training fits or which option is cheaper to finish. NVIDIA's A100 data sheet, H100 specifications and 4090 specifications are hardware references, not rental quotes or application benchmarks.

For a 100M-parameter dense training model, the illustrative 16-byte state representation is 1.6 GB before activations. For 5B it is 80 GB. Thus “any model below 5B fits full training on a 24 GB 4090” is false. A 7B LoRA or QLoRA adaptation is a different memory problem from full pretraining. A100-80 and A100-40 do not imply equal measured application speed even if a particular arithmetic peak matches. A faster expensive device can be cheaper per completed experiment, but that must follow from measured time and a timestamped quote, not vendor rankings.

Compare provider contracts, not an invented price table

Fill a worksheet for Runpod, Vast.ai, Modal, Lambda, CoreWeave, Nebius, Together, AWS and GCP only where you are considering them. Record exact service/product, quote time, region, device variant and count, CPU/RAM, local disk, persistent storage, network/egress, minimum billing unit, commitment, interruption terms, permitted data and teardown semantics. A marketplace offer is host-specific; a cloud brand is not a guarantee of quality, availability or a particular network. Do not equate dedicated cluster rental with a managed fine-tuning API: Together-style offerings must be compared by the exact product contract, export/checkpoint rights and model/data constraints.

For a persistent pod you control an ordinary process lifecycle but may pay while idle. A function platform can bind computation to function execution, yet timeouts, retries, warm containers, retained volumes and detached jobs still need a budget. Spot/interruptible capacity lowers a quoted rate in exchange for availability risk; on-demand does not mean hardware cannot fail. Enterprise IAM, compliance, support and existing credits can justify a hyperscaler. “Never use AWS” and “the cheapest provider always wins” are not engineering decision rules.

No historical price matrix is presented as current. The former named-model dollar totals mixed approximate FLOPs, invented utilization, vendor prices and incompatible reported training-hour boundaries. Replace that exercise with a transparent cost reconstruction: record a paper's reported device-hours, the stages included, and an explicit hypothetical rate; do not call the multiplication the lab's actual expenditure.

Cost worksheet and checkpoint cadence

Suppose a local/approved pilot measured 20,000 training tokens/sec and the job has 100 million tokens left. Training time is 5,000 seconds = 1.389 hours. At a hypothetical 2 currency units per GPU-hour, that component costs 2.778 units. Add startup, evaluation, checkpoint writes, idle time, retries, storage, egress and tax separately. A billing alert is not necessarily a hard cap.

For a dense 350M model and an illustrative 20 tokens/parameter allocation, D=7B tokens and 6ND=1.47e19 FLOPs. At an assumed sustained 124.8 TFLOP/s, training arithmetic is 32.719 GPU-hours. A 30% compute reserve gives 42.535 GPU-hours before storage and other overhead. Four devices would ideally divide wall time by four, not GPU-hours, but actual scale inefficiency changes both. This answers the original budget stretch without inventing a “cheapest” market quote.

For the original named-model exercise, the April Llama 3 model card, accessed 2026-10-08, reports 1.3M H100-80GB GPU-hours for 8B pretraining. Multiplying by a clearly hypothetical 2 units/GPU-hour gives 2.6M compute-only units. It does not include a known cloud quote, all prior experiments or your proposed cluster's efficiency, and dividing by an unrelated headline cluster size does not reveal elapsed time. This is the worked historical-receipt calculation, not a promise to reproduce model quality or Meta's real bill.

With checkpoint write time C minutes, useful-compute interval T, and approximately memoryless mean interruption time M, fractional overhead is approximately C/T + T/(2M), excluding restart time. Differentiate: -C/T² + 1/(2M)=0, so T*=sqrt(2CM). For C=.5 and M=120, T*=10.954 minutes and the two contributions are each about .04564, total .09129. This is a rare-failure approximation, not a guarantee; correlated reclamations and checkpoint failures invalidate it. At twice T*, overhead becomes 1.25× the optimum, not literally free. If mean interruption time is shorter than the checkpoint interval, completion may become impractical, but is not mathematically impossible under every distribution.

# CPU worksheet; rates are caller-supplied assumptions, never live quotes.
import math
 
def checkpoint_interval(write_minutes, mean_interruption_minutes):
    if write_minutes <= 0 or mean_interruption_minutes <= 0:
        raise ValueError("positive times required")
    return math.sqrt(2 * write_minutes * mean_interruption_minutes)
 
def budget(tokens, tokens_per_second, rate, extra_hours=0., storage_cost=0.):
    if tokens < 0 or tokens_per_second <= 0 or min(rate, extra_hours, storage_cost) < 0:
        raise ValueError("invalid budget input")
    hours = tokens / tokens_per_second / 3600
    return {"training_hours": hours,
            "assumed_total": (hours + extra_hours) * rate + storage_cost}
 
assert abs(budget(100_000_000, 20_000, 2)["training_hours"] - 1.3888888889) < 1e-8
assert abs(checkpoint_interval(.5, 120) - math.sqrt(120)) < 1e-12

Runpod reference: provision, inspect and connect

The Pod management documentation, storage types and SSH contract govern this walkthrough. Labels can change; verify them in the authenticated console before spending.

1. Account and budget: review funding, auto-refill and billing limits yourself. Write a maximum total cost and stop deadline, identify the console/API operation that terminates the Pod, and record which resources retain charges afterwards. Do not infer that an arbitrary prepaid balance or alert enforces the desired cap.
2. Template and storage: select an official/trusted image by immutable digest with a tested Python/PyTorch/CUDA lock. Do not copy an old runpod/pytorch tag and presume it still exists. Read the image's startup/exposed-port settings, attach the intended persistent volume at creation, and record region/mount identity. Do not expose a notebook or training dashboard publicly merely for convenience.
3. Spot or on-demand: choose interruptible only after a local resume drill. Record actual interruption/availability terms, including what survives reclamation; do not promise five seconds' warning or automatic reacquisition. On-demand still needs checkpoints.
4. Deploy and inspect: after an explicitly authorized purchase, verify the allocated device, volume mount, free space and driver/runtime. If any differ from the manifest, stop before training and correct the resource choice. No five-minute completion promise is warranted.
5. SSH: add only the public SSH key to provider credentials; retain the private key locally. Use the connection command returned for that Pod and verify the host key through an independent trusted route. Basic proxy SSH and full SSH with a public IP have different SCP/SFTP support; select the documented transfer-capable route when rsync is required.

# Reference only. Set these non-secret endpoint values from the authorized Pod.
: "${POD_HOST:?set documented endpoint}" "${POD_PORT:?set documented port}"
ssh -p "$POD_PORT" -i "$HOME/.ssh/id_ed25519" "root@$POD_HOST"
# Inside the authorized Pod:
nvidia-smi
python -c 'import torch; print(torch.__version__, torch.version.cuda); print(torch.cuda.is_available())'
findmnt /workspace
df -h /workspace

Inspect actual free memory after a full dry training step, not just nvidia-smi before Adam state allocation. If the driver cannot support the selected runtime, choose a compatible reviewed image; do not improvise root-level driver changes on a managed host. Keep the spin-up-to-first-step drill as a timing sheet (allocation, SSH, transfer, environment, first update), not a paid exercise required by this review.

Transfer code, data caches and the manifest

Git is appropriate for a clean reviewed revision: clone only the authorized repository and check out the recorded commit. A private repository requires its supported scoped authentication flow, not a token in a clone URL. A lock file and wheel/image digest belong in the manifest. Installing dependencies on the rental is a cost and trust decision; build the environment beforehand where possible.

For incremental transfers, first inspect rsync's dry-run file list. A trailing slash copies directory contents. Exclude private keys, .env files, VCS history if unnecessary, local caches and unrelated private data. Never use --delete as a default on a persistent checkpoint volume.

# Reference from the local machine; reviewed code only.
rsync -avzn --exclude='.git' --exclude='.env*' --exclude='secrets/' \
  --exclude='__pycache__/' --exclude='.venv/' \
  -e "ssh -p $POD_PORT -i $HOME/.ssh/id_ed25519" \
  ./reviewed-training/ "root@$POD_HOST:/workspace/training/"
# Remove only the dry-run n after reviewing the list and approving this transfer.

Runpod storage types distinguish container disk (lost on stop/restart), volume disk (retained until the Pod is deleted), and network volume (exists independently). /workspace alone does not prove independent persistence: a network volume can replace the ordinary volume disk there. Check the actual attached resource ID and mount. A network volume is not an independently verified backup, and may keep charging after compute termination.

Hugging Face Hub 0.34.4 documents that HF_HOME stores both token and cache by default. Do not put token-bearing home directories on a shared volume. Keep private authentication on the supported private credential path and direct only cache directories to authorized persistent storage:

# Reference environment for this process, not a blind shell-rc edit.
export HF_HUB_CACHE=/workspace/cache/hub
export HF_DATASETS_CACHE=/workspace/cache/datasets
# In an offline audited run, refuse network model/data resolution:
export HF_HUB_OFFLINE=1
export HF_DATASETS_OFFLINE=1

Pre-tokenized shards need corpus revision/license, tokenizer hash, preprocessing configuration, split definition, byte count and checksum. Verify those after transfer and before reusing a cache. Cached does not mean correct, permitted or secret-free. No corpus is downloaded for this lesson.

Launch once, keep failures visible and monitor remotely

Use a named terminal multiplexer session for interactive disconnect tolerance, or the provider's documented detached job path. A terminal multiplexer survives an SSH disconnect, not host loss or Pod termination. The following shell is a reference for a pre-existing reviewed train.py whose CLI implements these arguments. Unlike the old recipe it neither hides stderr nor converts failures into success.

# Reference inside the approved Pod, after environment and manifests are checked.
set -euo pipefail
cd /workspace/training
mkdir -p /workspace/run-output
sha256sum -c inputs.sha256
# Start tmux separately: tmux new -s training
# Choose the already-validated resume checkpoint in run.json; first run omits it.
python train.py --config run.json 2>&1 | tee /workspace/run-output/train.log
# pipefail preserves a failing Python exit code instead of silently claiming success.

The training program must have a hard update/token limit and wall-clock stop policy, an explicit resume path, atomic checkpoint publishing under the storage contract, and final success metadata written only after artifact validation. An unconditional infinite restart loop can repeatedly bill for an OOM or corrupt input. Classify failures, cap retry attempts and budget, and make “completed,” “interrupted” and “failed” distinct states. A local poweroff inside a container is not a proven provider-termination operation and may not stop the bill.

W&B's tracking interface provides init and log; configure privacy and supported authentication outside source code. Use offline mode for a no-network rehearsal, then opt into online logging only after reviewing what data leaves the machine. Log no secrets or raw private prompts by default. A browser on your phone can view the private run with your account, but telemetry is not a watchdog guarantee.

# Reference instrumentation: wandb 0.18.5 API; values come from your loop.
import wandb
 
def start_tracking(config):
    return wandb.init(project="training-audit", config=config, mode="offline")
 
def record_update(run, step, loss, lr, grad_norm, tokens_per_second, eval_loss=None):
    values = {"train/loss": float(loss), "train/lr": float(lr),
              "train/grad_norm": float(grad_norm), "train/tokens_per_second": float(tokens_per_second)}
    if eval_loss is not None:
        values["eval/loss"] = float(eval_loss)
    run.log(values, step=step)
# Caller uses try/finally to call run.finish(exit_code=0 or 1) for the actual outcome.

Inspect train/eval loss together, LR and grad norm around update boundaries, processed-token counts, throughput after warmup, host/data wait, allocated/reserved device memory and checkpoint age. GPU utilization alone can be high during a stuck/retrying computation; establish a progress heartbeat from successful updates. If a dashboard stops advancing, inspect the job and provider state rather than assuming the mobile browser is merely stale.

Modal reference: function lifecycle and durable output

A Modal Function specifies its image, accelerator and timeout instead of asking you to manage a long-lived SSH session. The GPU guide documents gpu="A100-40GB"; the Volume guide distinguishes commits/reloads from an ordinary shared filesystem. A snapshot can be stale in another container until reloaded. Volume retention is separate from function completion.

This complete remote wrapper invokes an existing reviewed local training script. It is an unexecuted service reference using the Modal SDK 1.1.4 wrapper API, cross-checked against its tagged image implementation and the service documentation snapshot of 2026-10-08; record the SDK lock and image digest in any actual reproduction. PyTorch is explicitly pinned in the image example. train.py and its config/data contract are inputs to the wrapper, not secretly supplied model downloads.

# Reference only: do not run modal run without authorizing remote resource use.
import modal
 
app = modal.App("reviewed-training")
volume = modal.Volume.from_name("reviewed-training-data", create_if_missing=False)
image = (modal.Image.debian_slim(python_version="3.11")
         .pip_install("torch==2.9.0")
         .add_local_file("train.py", "/app/train.py"))
 
@app.function(image=image, gpu="A100-40GB", timeout=4*60*60,
              volumes={"/data": volume})
def train():
    import subprocess
    # Config references already-staged permitted inputs and a finite run budget.
    try:
        subprocess.run(["python", "/app/train.py", "--config", "/data/run.json"], check=True)
    finally:
        # Persist completed checkpoints/logs even on a script failure.
        # A hard container kill can prevent finally; checkpoint/commit periodically too.
        volume.commit()
 
@app.local_entrypoint()
def main():
    train.remote()

For a job intended to survive local disconnection, the documented command is modal run --detach modal_train.py; plain modal run is not interchangeable. The CLI reference explicitly describes --detach. The finite function timeout bounds that invocation, not unlimited retries or all storage costs. Record the app/run ID, monitor the dashboard, inspect completion status and retained resources. Do not promise that “you cannot leave anything billed by accident.”

Interruption drill and four-stage teardown

A resumable checkpoint includes model, optimizer, scheduler/scaler, random states, processed-token/update counts, data cursor/sampler epoch and configuration hashes. Publish a complete generation plus a manifest and keep one prior validated generation. Under local filesystem semantics, write a temporary file, flush as required and rename; remote/object storage requires its own commit protocol. On resume compare the next update to uninterrupted execution, not merely whether load returned successfully.

Before relying on spot, simulate interruption locally at a checkpoint boundary and mid-update. The mid-update case should restart from the last complete update, not reuse half-applied state. Track lost work, restore time and reacquisition separately. The square-root cadence is a planning approximation; measured recovery decides feasibility.

1. Seal the result: let a complete update finish or record an interrupted state; save the latest complete checkpoint, verify checksum and loadability, and record final or partial status. Timestamps alone are insufficient.
2. Close telemetry and copy independently: finish logging with the true exit state; copy permitted artifacts to independent authorized storage and verify hashes/load there. Successful rsync exit alone does not prove the model can resume.
3. Terminate the intended compute resource: use its provider console/API, match the resource ID, and confirm the resulting provider state. Closing SSH, pressing Ctrl+C, poweroff, stopping a process or writing DONE is not this step.
4. Reconcile everything billable: list remaining volumes, snapshots, functions/jobs, reservations and storage charges. Retain only deliberate resources, with owner, cost and expiry. A useful network volume may intentionally remain; do not destroy it just to claim a zero bill.

Failure cases and worked retention answers

Forgotten Pod: a finished or crashed process can coexist with running compute. A deadline reminder plus provider-state verification and an authorized provider-side stop mechanism is stronger than a sleeping shell. Cache fills root disk: inspect mount/free bytes, move only caches to the correct volume and rerun checksum validation; do not restart into the same disk-full loop. SSH closes: a detached process may continue billing. Spot disappears: no shell on the vanished host can reacquire capacity; a bounded external orchestration/console action is a separate system. Logs show success, checkpoint corrupt: the success marker was published too early.

1. What does a 100M/2B-token run cost? Approximately 1.2e18 dense-model FLOPs, but no dollar answer without measured sustained throughput, overhead and a quote. The former fixed three-dollar answer is withdrawn, not a learning objective retired.
2. Which GPU should I choose? Eliminate configurations that cannot fit the exact peak; compare expected completion cost and deadline for the survivors. For a model with 30 GB measured peak, a 24 GB 4090 cannot run that unchanged configuration, whatever its advertised arithmetic throughput.
3. Why network volume? It is independent of Pod lifetime under the documented service contract. Verify attachment and retain a separate tested copy; a Pod volume disk and container disk have different lifetimes.
4. When Modal rather than a Pod? A debugged finite function with an explicit image/output contract can simplify unattended work. Interactive iterative debugging may suit a Pod. Neither choice has an inherent universal price premium or guarantees no idle/storage charges.
5. How do I survive interruptions? Complete-state checkpoints, proven continuation and bounded classified retries. “Every 500 steps” is not a time interval without measured step time; infinite restart is not fault tolerance.
6. Reconstruct a published run's cost. If a source reports H device-hours, a hypothetical rate r gives H×r compute-only units. State hardware variant, stages and exclusions. Do not replace the reported token budget with 20N and call it a reproduction.
7. Two habits? Measure time-to-completion before scaling; independently verify both the saved artifact and the final billing/resource state.

Spaced review: tomorrow reconstruct the manifest, cache/credential separation and four-stage teardown without notes. In a week solve the 350M worksheet using a new hypothetical rate and an observed interruption model. Bring back a resource ID, a tested checkpoint contract and a budget equation—not an old spot price. Next: a bounded pretraining experiment uses this operating discipline; this lesson does not claim you already ran it.

Previous: ← DL S051 · Next: DL S053 →

Original optional resource links

The original optional video/reading destinations are retained here for continuity, not as current API, price, benchmark or publication-date evidence. The versioned primary citations in the teaching above govern the corrected claims. These supplementary destinations were not all fetched or watched; a missing or changed resource does not replace the local derivations, procedures and answers.

• https://www.youtube.com/watch?v=hBBOjCiFcuo
• https://www.youtube.com/watch?v=Sy1psHS3w3I
• https://docs.runpod.io/pods/overview
• https://modal.com/docs/guide/gpu
• https://www.runpod.io/pricing
• https://vast.ai
• https://modal.com/docs/examples/llm-finetuning
• https://lambdalabs.com/service/gpu-cloud
• https://github.com/karpathy/llm.c/discussions/677