Adaptation with LoRA: low-rank updates and masked supervision

What LoRA changes, and what it leaves expensive

Deep Learning & LLMs From Scratch · Session 60 / 80 · M10 · Fine-tuning + alignment.

Pretraining estimates patterns in a broad corpus; supervised adaptation changes behavior on selected demonstrations. LoRA changes which parameters represent an update, not the loss, truth of the data, licensing requirements or safety of the result. A frozen base can still produce very different, even degraded, outputs when a learned adapter is added.

Prerequisites are linear layers/backpropagation, rank and an SFT loop with a well-defined response mask. This lesson derives gradients and rectangular counts, implements and injects a layer, tests merge parity, gives a pinned local-checkpoint PEFT/TRL reference, and distinguishes rsLoRA, DoRA, PiSSA and LoRA+. The small NumPy and PyTorch programs run on CPU in the existing review environment. No pretrained weights, dataset, GPU fine-tune or service benchmark is executed.

A 7B model's BF16 weight payload alone is approximately 14 decimal GB, not 14 GiB and not its training peak. Full adaptation also needs trainable gradients/optimizer state and activations. Freezing most weights removes much of the trainable-state bill, but not the base weights or all backward activations. Thus LoRA does not guarantee that every 8B SFT setup fits 24 GB or matches full fine-tuning within one point.

Factor the update, not the original matrix

Using row-vector inputs, let the frozen base matrix be $W_0\in\mathbb R^{d_{in}\times d_{out}}$. Train $A\in\mathbb R^{d_{in}\times r}$ and $B\in\mathbb R^{r\times d_{out}}$:

The column-vector/PyTorch weight-storage convention instead uses W0 (out,in), A (r,in), B (out,r) and delta W=sBA. Both describe the same update after transposition. Only the update has rank at most r; the base may be full rank. Hu et al., v2, section 4.1 propose this low-rank adaptation hypothesis; it is not a theorem that every useful fine-tune delta has low rank.

The adapter count is r*(in+out), versus in*out base weights. At 4096×4096 and r=8 this is 65,536 versus 16,777,216, a 256× trainable-count ratio. A rectangular value projection (1024,4096) needs only 40,960 adapter parameters. For a stated 32-layer GQA configuration with those Q/V widths, total Q/V count is 32*(65,536+40,960)=3,407,872. The old 4,194,304 calculation applies to two square 4096 projections, not a GQA model with narrower V. Inspect actual module shapes before attributing a count to Llama, Mistral or SmolLM.

Initialization and a hand calculation

With upstream row-batch gradient G, matrix calculus gives

Random A and zero B make the starting update zero, the first A gradient zero, and the B gradient potentially nonzero. Both zero traps plain gradient descent at the origin. Zero A with random B is another output-preserving, trainable asymmetric initialization; the conventional choice is not mathematically unique. Nonzero factors can even multiply to zero in special subspaces. Neither randomness nor a small adapter proves a safe learning-rate range.

Take x=[1,1], W0=I, A=[[1],[2]], B=[[0,0]], s=2 and target [2,0] under half squared error. The initial output is [1,1], G=[-1,1], dB=[-6,6], dA=0. At learning rate .1, B becomes [.6,-.6]; the next output is [4.6,-2.6], an overshoot. For x=[2,-1], xA=0 and B's gradient would also vanish for that sample. This is why “one zero factor guarantees a nonzero first step” needs a data/gradient qualification.

For the input derivative, the frozen path contributes G@W0.T. Earlier adapters need it. requires_grad_(False) on base parameters avoids their parameter gradients; enclosing a whole frozen network in no_grad() can sever the path to trainable modules earlier in the graph.

Train and merge a toy adapter

The exact retained program is a regression experiment on an explicitly rank-one target correction. It tests the implementation, not language-model quality, and computes both factor gradients from the same parameter snapshot before updating either.

import numpy as np
rng = np.random.default_rng(4)
X = rng.normal(size=(20, 3))
W0 = rng.normal(size=(3, 2))
base_copy = W0.copy()
target_update = np.array([[1.], [-.5], [.25]]) @ np.array([[.4, -.2]])
y = X @ (W0 + target_update)
A = rng.normal(0, .1, (3, 1)); B = np.zeros((1, 2))
s = 1.
np.testing.assert_array_equal(X @ W0 + s*(X@A)@B, X @ W0)
 
for step in range(2000):
    residual = X@W0 + s*(X@A)@B - y
    g = residual / len(X)  # loss is half squared sum over outputs, mean over rows
    dB = s*(X@A).T @ g
    dA = s*X.T @ g @ B.T
    if step == 0:
        assert np.all(dA == 0) and np.linalg.norm(dB) > 0
    A -= .1*dA
    B -= .1*dB
unmerged = X@W0 + s*(X@A)@B
merged = X @ (W0 + s*A@B)
np.testing.assert_allclose(unmerged, merged, atol=1e-12)
np.testing.assert_array_equal(W0, base_copy)
assert np.mean((unmerged-y)**2) < 1e-5
print('adapter MSE', np.mean((unmerged-y)**2), 'merge parity verified')

Merge parity is exact real arithmetic for the deterministic linear update and approximate floating-point equality in a particular dtype. It does not establish speedup or quality for quantized/nonlinear variants.

A minimal PyTorch module and its contract

The earlier “25 lines including whitespace” referred to the layout of its particular illustrative module, not an implementation contract or a completeness test. This version is governed by the explicit tensor, gradient and integration requirements below; line count does not establish those properties.

The exact protected module below inherits the base tensor's device/dtype. In this review it is exercised with tiny CPU tensors on PyTorch 2.9.0+cpu. It omits dropout, tied-weight handling, quantization and tensor parallelism deliberately; those remain explicit requirements for a broader library integration, not invisible working features.

import math
import torch
from torch import nn
 
class LoRALinear(nn.Module):
    def __init__(self, base, rank=8, alpha=16):
        super().__init__()
        if rank <= 0:
            raise ValueError("rank must be positive")
        self.base = base
        self.base.requires_grad_(False)
        self.scale = alpha / rank
        self.A = nn.Parameter(base.weight.new_empty(rank, base.in_features))
        self.B = nn.Parameter(base.weight.new_zeros(base.out_features, rank))
        nn.init.kaiming_uniform_(self.A, a=math.sqrt(5))
 
    def forward(self, x):
        return self.base(x) + self.scale * ((x @ self.A.T) @ self.B.T)

For an input batch (...,in), x@A.T has shape (...,r), then multiplication by B.T returns (...,out). The base bias remains in base(x). Freezing only the wrapped linear is insufficient if the rest of the original model remains trainable.

Inject without mutating an active traversal

Collect targets before replacement and freeze the whole base first. This helper uses the protected class above, checks for an empty or repeated selection, and verifies a complete tiny model rather than printing an invented trainable count.

# CPU continuation of the protected LoRALinear definition above.
def inject_lora(model, target_names=("q_proj", "v_proj"), rank=8, alpha=16):
    model.requires_grad_(False)
    targets = []
    for parent in model.modules():
        for name, child in parent.named_children():
            if name in target_names:
                if not isinstance(child, nn.Linear):
                    raise ValueError("target is not a fresh Linear")
                targets.append((parent, name, child))
    if not targets:
        raise ValueError("no requested target modules")
    for parent, name, child in targets:
        setattr(parent, name, LoRALinear(child, rank=rank, alpha=alpha))
    return model
 
torch.manual_seed(5)
model = nn.ModuleDict({"q_proj": nn.Linear(4, 4), "v_proj": nn.Linear(4, 2)})
x = torch.randn(3, 4, requires_grad=True)
base_outputs = {k: layer(x).detach().clone() for k, layer in model.items()}
inject_lora(model, rank=2, alpha=4)
assert sum(p.numel() for p in model.parameters() if p.requires_grad) == 28
for name, layer in model.items():
    torch.testing.assert_close(layer(x), base_outputs[name])
loss = sum(layer(x).square().mean() for layer in model.values())
loss.backward()
assert x.grad is not None  # the frozen path still transmits input gradients
assert all(layer.base.weight.grad is None for layer in model.values())
assert all(torch.count_nonzero(layer.A.grad) == 0 for layer in model.values())
assert any(torch.count_nonzero(layer.B.grad) > 0 for layer in model.values())

Injection precondition: the tiny helper assumes an unshared module tree. If q_proj and v_proj refer to the same nn.Linear object in one parent, PyTorch's named_children() deduplicates it: this helper wraps q_proj but can leave v_proj pointing to the unwrapped base. Shared aliases therefore need an identity-aware replacement policy and parity tests before injection; this example does not implement that policy. It also freezes the base before validation, so an exception is not a transaction rollback—start with a fresh model after a failed attempt.

Real model families use different module names, fused projections and tied weights. Print actual matched full names, shapes and trainable parameters; compare base bytes before/after an update. Do not mutate shared modules twice or silently leave embeddings/norms trainable. An optimizer should receive only the intended trainable parameters.

Rank, alpha, placement and learning rate

Rank controls a matrix-rank upper bound, not a count of new facts. Alpha/r scales the update and both factor gradients. Setting alpha=2r gives scale two across ranks, not identical training dynamics: factor initialization, norms, the number of directions and the optimizer still change. A rank/LR/alpha sweep is an experiment, not a universal r=8 or “10–100× SFT LR” rule.

A useful controlled comparison is Q/V-only versus all supported linear layers at similar trainable-parameter budgets. Increasing rank on Q/V differs from adding rank to MLP or output projections. Domain/style knowledge is not exclusively stored in MLPs, and attention is not exclusively factual routing. Choose using held-out domain improvement, regression evaluation and actual memory. A loss decrease on demonstrations is not enough.

Count memory as frozen weights + adapter weights/gradients/optimizer/master state + saved activations + temporary kernels/reserve. Gradient checkpointing relieves some activation storage at a compute cost. QLoRA additionally quantizes the frozen base; it is not merely “higher-rank LoRA.” Neither method removes the need to review data provenance or evaluate forgetting.

Reference procedure: measure a forward/backward window on one CUDA device. This is a PyTorch 2.9.0 API reference, UNEXECUTED on CUDA. Its exact control flow is tested only with a CPU state-machine stub; the stub's invented bytes are not GPU measurements. Governing primary APIs are memory_allocated, reset_peak_memory_stats, max_memory_allocated and reserved counters and synchronize. memory_allocated counts current tensor allocations; memory_reserved counts memory managed by the caching allocator. Neither is the whole device usage shown by nvidia-smi.

For a separately authorized run, construct the approved dense, unsharded model, batch and any optimizer before this window, all on the explicitly indexed device, for example torch.device('cuda:0'). Declare dtype, batch/sequence shapes, attention implementation, checkpointing, train/eval mode, and whether optimizer state or warmup already exists. Release warmup losses/graphs; otherwise the baseline includes them. Do not run concurrent allocating work on this device. The callable must create a fresh scalar loss from a forward pass each time, not return a retained graph or cached loss.

# Reference only: no call to real CUDA is made in this lesson's CPU checks.
# Supply the real torch module ONLY in a separately authorized CUDA experiment.
def measure_backward_peak(torch_api, model, forward_loss, device):
    device = torch_api.device(device)
    if device.type != 'cuda' or device.index is None:
        raise ValueError('explicit indexed CUDA device required')
    if not torch_api.cuda.is_available():
        raise RuntimeError('CUDA unavailable; do not report a measurement')
    parameters = list(model.parameters())
    tensors = parameters + list(model.buffers())
    if not parameters or any(t.device != device for t in tensors):
        raise ValueError('all model tensors must be on the measured device')
    frozen_payload = sum(p.numel() * p.element_size() for p in parameters
                         if not p.requires_grad)
    trainable_count = sum(p.numel() for p in parameters if p.requires_grad)
    model.zero_grad(set_to_none=True)
    cuda = torch_api.cuda
    cuda.synchronize(device)
    baseline = cuda.memory_allocated(device)
    baseline_reserved = cuda.memory_reserved(device)
    cuda.reset_peak_memory_stats(device)
    loss = forward_loss()
    if loss.ndim != 0 or loss.device != device:
        raise ValueError('fresh scalar loss on the measured device required')
    loss.backward()
    cuda.synchronize(device)
    peak = cuda.max_memory_allocated(device)
    peak_reserved = cuda.max_memory_reserved(device)
    live_after = cuda.memory_allocated(device)
    reserved_after = cuda.memory_reserved(device)
    return {'frozen_parameter_payload_bytes': frozen_payload,
            'trainable_parameters': trainable_count,
            'baseline_allocated_bytes': baseline,
            'baseline_reserved_bytes': baseline_reserved,
            'peak_allocated_bytes': peak,
            'peak_minus_baseline_bytes': peak - baseline,
            'live_after_backward_bytes': live_after,
            'peak_reserved_bytes': peak_reserved,
            'reserved_after_backward_bytes': reserved_after}

Order matters: clear prior parameter gradients → synchronize prior work → record baseline → reset peaks → forward → backward → synchronize completion → read peaks and current counters. Resetting after the forward would erase forward activation peaks; omitting reset can report an earlier iteration. Synchronization does not clear memory. No empty_cache() is inserted to make a result look smaller, and clearing peaks does not release the model or allocator cache.

The baseline includes the already resident base/adapter, inputs, buffers and any pre-existing optimizer state; it excludes old gradients cleared by set_to_none=True. The absolute peak is a live-tensor high-water mark since reset, while peak-minus-baseline is the increase over this particular starting state, not “activation bytes.” At the post-backward sample the local loss still exists; autograd normally releases saved intermediates, but parameter gradients remain. A large transient peak can exceed the final live allocation. Reserved peak is a different high-water mark; do not add it to allocated peak. These maxima may occur at different times.

numel()*element_size() is a logical frozen-parameter payload for ordinary dense, non-overlapping storage, not allocated VRAM: buffers, shared-storage aliases, quantization wrappers, padding, optimizer state, CUDA context and non-PyTorch allocations can change the comparison. This window omits optimizer.step() and first-step optimizer-state allocation; a full-step experiment must deliberately include those calls after backward and before the final synchronization/read. Record repeated windows and warmup policy rather than claiming one number proves that a 7B model fits. Practice: explain why resetting after backward or substituting reserved bytes for tensor bytes invalidates the stated measurement contract. Answer: the first loses the interval's peak; the second changes the quantity rather than measuring the same memory more accurately.

Supervised target masks are part of the algorithm

For a prompt-response pair under response-only supervision,

The denominator counts selected target tokens, not padded length. Prompt tokens can influence responses through attention even if their own prediction loss is masked. Reject an empty response mask. Include EOS deliberately; if PAD and EOS share an ID, masking every occurrence of that ID accidentally removes real EOS targets. PyTorch 2.9 CrossEntropyLoss specifies ignored class-index targets and reduction semantics; weighted losses can have a different denominator.

Packing changes attention and boundaries. The reference below disables packing to make prompt/response alignment inspectable. A production packed run needs its own boundary/position/mask verification. Tokenizer and chat-template changes invalidate cached tokenized examples.

PEFT and TRL: complete local-checkpoint reference

This is a source-reviewed, unexecuted GPU reference, not a claim that Llama-3-8B fits a single 24 GB GPU. Use a separately authorized, licensed local base/tokenizer directory with a pinned revision (an 8B Llama checkpoint if that is your approved experiment), and a reviewed local JSONL dataset with prompt and response strings. No code here downloads them. The historical API contract is Transformers 4.45.2, TRL 0.11.4, PEFT 0.13.2, datasets 3.0.1, plus a compatible PyTorch/CUDA lock chosen and tested by the operator. The newer PEFT 0.17 survey below is not silently substituted into this lock.

TRL 0.11.4 SFTTrainer accepts tokenizer and the SFTConfig defines max_seq_length and dataset preparation controls. Newer APIs can use different argument names.

# Reference only. Inputs are local approved artifacts, no downloads.
import json
import os
import torch
from datasets import Dataset
from transformers import AutoModelForCausalLM, AutoTokenizer, DataCollatorForSeq2Seq
from peft import LoraConfig, get_peft_model
from trl import SFTConfig, SFTTrainer
 
base_path = os.environ["LOCAL_BASE"]
data_path = os.environ["LOCAL_SFT_JSONL"]
tok = AutoTokenizer.from_pretrained(base_path, local_files_only=True, trust_remote_code=False)
if tok.chat_template is None:
    raise ValueError("record and supply the model's reviewed chat template")
if tok.pad_token_id is None:
    tok.pad_token = tok.eos_token
if tok.pad_token_id is None:
    raise ValueError("tokenizer needs a declared padding token")
tok.padding_side = "right"
records = []
with open(data_path, encoding="utf-8") as f:
    for line in f:
        row = json.loads(line)
        user = [{"role": "user", "content": row["prompt"]}]
        prefix = tok.apply_chat_template(user, tokenize=True, add_generation_prompt=True)
        full = tok.apply_chat_template(user + [{"role": "assistant", "content": row["response"]}],
                                       tokenize=True, add_generation_prompt=False)
        if full[:len(prefix)] != prefix or len(full) <= len(prefix):
            raise ValueError("template prefix mismatch or empty response")
        if len(full) > 1024:
            raise ValueError("overlong example; review truncation rather than silently cut")
        records.append({"input_ids": full, "attention_mask": [1]*len(full),
                        "labels": [-100]*len(prefix) + full[len(prefix):]})
if not records:
    raise ValueError("empty SFT dataset")
model = AutoModelForCausalLM.from_pretrained(base_path, local_files_only=True,
                                           trust_remote_code=False, torch_dtype=torch.bfloat16)
model.config.use_cache = False
cfg = LoraConfig(r=8, lora_alpha=16, target_modules=["q_proj", "v_proj"],
                 lora_dropout=0.05, bias="none", task_type="CAUSAL_LM")
model = get_peft_model(model, cfg)
model.print_trainable_parameters()  # inspect names and counts, not just this total
args = SFTConfig(output_dir="adapter-run", max_steps=20, learning_rate=2e-4,
                 per_device_train_batch_size=1, gradient_accumulation_steps=4,
                 max_seq_length=1024, packing=False,
                 dataset_kwargs={"skip_prepare_dataset": True},
                 bf16=True, logging_steps=1, save_steps=10, report_to=[],
                 remove_unused_columns=False)
trainer = SFTTrainer(model=model, args=args, train_dataset=Dataset.from_list(records),
                     tokenizer=tok,
                     data_collator=DataCollatorForSeq2Seq(tok, padding=True, label_pad_token_id=-100))
trainer.train()
trainer.save_model("adapter-run/final")
tok.save_pretrained("adapter-run/final")

The learning rate and 20 updates are smoke-test choices, not recommended convergence hyperparameters. Before a real fine-tune, prompt-level split/decontamination, data rights, decoded masks, a one-batch overfit diagnostic, base-versus-adapter held-out comparisons and out-of-domain regression tests must pass. Record exact base and tokenizer hashes, modules/rank/scaling/dropout, precision, software lock, training token budget, split and evaluation settings. Adapter weights are not a standalone model.

Loading and merging the saved adapter

# Reference continuation with the same pinned stack and local artifacts.
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained(base_path, local_files_only=True,
                                          trust_remote_code=False, torch_dtype=torch.bfloat16)
adapted = PeftModel.from_pretrained(base, "adapter-run/final", is_trainable=False)
adapted.eval()
merged = adapted.merge_and_unload()
merged.save_pretrained("adapter-run/merged")
# Keep the original base identity AND adapter-run/final, not just merged weights.

Test unmerged versus merged logits before accepting the artifact, using deterministic evaluation, the same prompts/dtype and a declared numerical tolerance. merge_and_unload removes adapter structure from the returned ordinary model; preserve the adapter for reconstruction or continuing that parameterization. It does not certify that every inference server supports the base architecture or tokenizer.

Merge, quantization and multi-adapter serving

Merging folds sBA into W once, avoiding the separate linear adapter matmuls. “No additional LoRA inference latency” refers to that algebraic construction, not zero latency overall or a universal 5–15% unmerged overhead. Keep adapters separate for per-request switching or when the validated computation uses a quantized base plus a higher-precision branch. Cache keys must include adapter identity; stale KV state from another adapter represents a different function.

Quantization is nonlinear: Q(W+delta) need not equal Q(W)+delta. A small delta within a quantization bin can disappear after merging/requantizing; one near a threshold can change the bin. There is no fixed INT4 step of 1/128 independent of scale/group. Do not add floating deltas directly into integer storage. Follow the supported dequantize/merge/requantize path and rerun task/long-context tests, or keep the validated high-precision adapter path. Retain both artifacts and base identity even when serving only a merged model.

Variants: identify the changed assumption

These methods are alternatives/controlled ablations, not a bundle of flags to turn on together. The PEFT 0.17.0 guide documents version-specific options; compatibility and quality still need tests.

rsLoRA: rank-dependent scaling

Kalajdzievski, 2023 v1 studies rank scaling and replaces alpha/r with alpha/sqrt(r). At fixed alpha=16 and r=64, the scales are .25 versus 2: a major changed update scale, not a free flag. It is not alpha=r², and no universal rank threshold guarantees a gain. Hold other knobs fixed for one comparison, then tune learning rate/alpha separately.

DoRA: magnitude and direction

Liu et al., 2024 v1 separate weight magnitude and direction, with a low-rank directional update and an additional trainable magnitude component. LoRA itself can change both magnitude and direction; the new parameterization controls them differently. Normalization and extra parameters alter compute/memory; “same memory, always 15% overhead” was not established. Evaluate supported layers, quantization and merge behavior in the chosen implementation.

Version-bound DoRA configuration — external integration UNEXECUTED. The governing PEFT v0.17.0 LoraConfig declaration declares use_dora: bool = field(default=False, ...); set use_dora=True to request the extra magnitude parameterization. This reference is deliberately separate from the PEFT 0.13.2 trainer lock above: it does not silently upgrade that environment, claim every version since 0.10 is equivalent, or combine DoRA with rsLoRA/PiSSA.

# PEFT 0.17.0 reference; no imports, model loads or integration calls run here.
# In an authorized environment: import peft; then pass peft and an approved base.
def configure_dora(base_model, peft_api):
    if peft_api.__version__ != '0.17.0':
        raise ValueError('this reference is bound to PEFT 0.17.0')
    config = peft_api.LoraConfig(
        r=8, lora_alpha=16, target_modules=['q_proj', 'v_proj'],
        lora_dropout=0.0, bias='none', task_type='CAUSAL_LM',
        use_dora=True)
    return peft_api.get_peft_model(base_model, config)

The caller must provide a fresh compatible causal-LM base with actual supported linear q_proj/v_proj modules and a validated dependency lock; inspect exact module matches and the added trainable magnitude parameters. The same-version guide lists embedding, linear and Conv2d support, whereas the config help mentions linear/Conv2D only. This reference stays within their linear-layer overlap instead of using that documentation discrepancy to promise embedding compatibility. The guide warns of extra overhead and reported QDoRA/DeepSpeed ZeRO-2 issues. Config validation also rejects use_dora with megatron_config and with lora_bias=True; bias='none' here is the separate base-bias training option, not lora_bias.

Zero dropout is an explicit comparison choice, not a guaranteed speedup. Match base/data/masks and tune one change at a time, inspect gradients and trainable names, then test save/reload and deterministic unmerged-versus-merged logits for the exact supported precision path before serving. DoRA adds normalization and magnitude parameters, so ordinary LoRA's protected toy merge does not prove DoRA merge parity. The CPU stub checks only the version gate, configuration arguments and handoff to get_peft_model; actual PEFT injection, supported architectures, optimizer behavior, quantized/distributed execution, quality and memory remain unexecuted. Practice: would toggling this flag in the older trainer snippet constitute a tested DoRA run? No: neither a matching dependency lock nor injection/training/merge validation follows from source configuration alone.

PiSSA: preserve the initial total weight

Meng et al., 2024 v1 initialize trainable factors from principal singular components and freeze the residual. If W=UΣVᵀ, take a rank-r component W_r and set W_res=W−W_r; initialize BA=W_r (accounting for the chosen scale) so their sum still equals W. Simply adding the SVD component to the unchanged whole W double-counts it. SVD cost depends on shapes and algorithm; faster convergence reported in selected experiments is not a guarantee for your task.

LoRA+: factor-specific learning rates

Hayou, Ghosh and Yu, 2024 v1 analyze using distinct rates for A and B. Their justification is feature-learning/scaling behavior, not solely that one factor starts at zero. A fixed ratio such as 16 is an experimental choice, not universal. Use explicit optimizer groups and verify each factor's actual learning rate; merely setting LoraConfig is not all of optimizer configuration.

A sensible experiment matrix begins with a known working ordinary LoRA run, changes one of scaling, parameterization, initialization or optimizer, and records the new trainable count, measured peak and held-out scores. The old “2025 default recipe” that enabled rsLoRA, DoRA and PiSSA together without a compatibility/ablation argument is not restored as advice.

Retention: worked answers and a rank experiment

1. State the rule and gradients. y=XW0+s(XA)B; dB=s*(XA).T@G, dA=s*X.T@G@B.T. In column-vector storage the delta is sBA. Do not mix shapes.
2. Why random A/zero B? It preserves the initial function while usually allowing B to learn first. The reversed asymmetric choice also works in principle; both zero is a trap. A particular input can still give zero gradients.
3. Why not reuse a claimed universal high LoRA LR? Factor/scaling/placement/data determine dynamics. The numeric overshoot above disproves safety from trainable count alone; frozen base bytes do not prevent behavioral forgetting.
4. Which modules first? Q/V-only is an economical candidate; wider attention/MLP coverage increases capacity and state. Compare actual matched shapes and task results. No strict style-versus-factuality partition justifies a universal placement rule.
5. What does the GQA counting exercise give? Under the stated Q (4096,4096), V (1024,4096), r=8, 32-layer shapes: 3,407,872. Under two square matrices: 4,194,304. Neither number is a total-memory ratio.
6. When not to merge? Multi-adapter switching, continued adapter-parameter training, or an unvalidated quantization transform. Always retain adapter plus compatible base identity; quantization sensitivity is not solely a bit-width threshold.
7. Measure intrinsic rank. On an authorized small full fine-tune, align matching base/final tensor names, compute delta in sufficient precision, SVD it, and measure rank-r approximation error. For singular values 5,2,1, rank one retains 25/30 squared Frobenius energy and leaves sqrt(5) error; rank two retains 29/30 and leaves 1. Then replace the delta with its truncated approximation and evaluate behavior. There is no guaranteed knee at rank 8 or 90% task accuracy from 90% energy.
8. Explain it to an SFT colleague. The same masked demonstration loss trains a restricted additive update while base tensors remain fixed. This saves much optimizer/gradient state, not all activation or base memory. Placement/rank restrict capacity; measured validation and regression tradeoffs decide usefulness. It is not RL, grounding or access control.

Spaced review: revisit linear shapes and SFT masks tomorrow; in a week repeat merge parity and the rectangular-count exercise from memory. QLoRA next reduces frozen-base storage using NF4, double quantization and paged-optimizer techniques; it does not make every configuration fit or every output safe. Bring back the equation, actual module list and a tested mask—not an unsupported universal learning rate.

Previous: ← DL S059 · Next: DL S061 →

Original optional resource links

The original optional video/reading destinations are retained here for continuity, not as current API, price, benchmark or publication-date evidence. The versioned primary citations in the teaching above govern the corrected claims. These supplementary destinations were not all fetched or watched; a missing or changed resource does not replace the local derivations, procedures and answers.

• https://www.youtube.com/watch?v=t509sv5MT0w
• https://www.youtube.com/watch?v=YVU5wAA6Txo
• https://arxiv.org/abs/2106.09685
• https://huggingface.co/docs/peft/en/conceptual_guides/lora