Supervised fine-tuning: objective, templates and local recipes

SFT changes a conditional distribution

Goal: derive masked next-token supervision, validate chat boundaries, run a complete bounded local SFT example, and specify a pinned local-only TRL recipe. SFT continues supervised optimization on demonstrations. It is not a new architecture, not PPO/DPO, and not synonymous with LoRA: full-parameter updates or adapters can optimize the same SFT objective.

A base model may continue a question with another question because it models the surrounding text distribution. Demonstrations can make an assistant response and stop condition more likely. But SFT is not incapable of learning facts or capabilities. “Mostly format” is a hypothesis studied in settings such as LIMA, not a universal theorem. SFT can learn/memorize new information, alter behavior and forget prior skills; generalization and factual reliability require independent tests. Calling it casting rather than education is at most a partial analogy.

Prompting is a cheap way to define desired behavior before training. Retrieval is often preferable for frequently changing, attributable facts; SFT can teach format and use of retrieved context. Neither guarantees correctness or access control. Choose among prompting, retrieval, SFT and combinations by held-out task failures, update frequency, latency and ownership cost—not a rigid facts-versus-style dichotomy.

Derive assistant-only loss and its gradient

For token sequence z and a label-position mask m, a causal logit at position t-1 predicts z_t. The assistant-only objective is

The denominator excludes ignored/padding targets and the unpredicted first position. With14 prompt tokens and response [5,EOS], two terms are supervised: the final prompt position predicts5, then5 predictsEOS. Shift logits/labels once. HuggingFace causal-LM loss usually shifts internally when passed same-length labels; do not pre-shift and let it shift again.

For a supervised position, derivative with respect to logits is (softmax(logits)-one_hot(target))/N; ignored target positions contribute zero direct loss. Prompt tokens remain input context, so their embeddings and earlier representations can receive gradients through assistant predictions. Loss masking does not remove prompt attention or freeze prompt embeddings.

PyTorch CE defaults to ignore_index=-100; it is an API convention, not the only legal sentinel across every framework/loss. A wrong negative target often raises rather than silently training random tokens. All-ignored batches must be rejected. Padding mask and label mask are distinct; if pad and EOS IDs coincide, mask by position/length, not every occurrence of that ID, or real EOS supervision disappears.

Worked weighting example

At exactly uniform predictions, masked and full-sequence mean CE are both ln(V), regardless of prompt length. For500 prompt tokens and50 response tokens, the full-sequence loss has499 prompt-target terms and50 response-target terms under the usual shift. Response-only optimizes just50 terms. This changes objective weighting, not a guaranteed11× convergence penalty. The sign of the measured loss change after masking depends on which tokens are easier; a loss decrease does not prove an inverted mask.

Whole-sequence instruction modeling is a legitimate alternative. Shi et al.'s Instruction Tuning With Loss Over Instructions studies when including instruction loss helps; it does not validate a universal “90% of tasks need masking” or a library-independent prompt_loss_weight=0.1 flag. Define a weighted sum explicitly, normalize the chosen units and ablate under a matched budget.

A complete bounded CPU SFT program

This program consumes already-tokenized segments with explicit assistant flags, then pads by length. It uses the Architecture Choices decoder for a generated-symbol fixture. There is no tokenizer download, Alpaca training or claim of natural-language instruction following. Save as sft_local.py beside architecture.py.

import copy
import torch
import torch.nn.functional as F
from architecture import Config,Decoder
 
def encode_segments(segments,eos,max_length,vocab):
    ids=[]; labels=[]
    if not 0<=eos<vocab or max_length<2: raise ValueError('encoding config')
    for tokens,assistant in segments:
        if not tokens or any(not isinstance(t,int) or not 0<=t<vocab for t in tokens):
            raise ValueError('invalid segment')
        part=list(tokens)+([eos] if assistant else [])
        ids.extend(part); labels.extend(part if assistant else [-100]*len(part))
    # Do not silently train on a truncated answer or missing stop token.
    if len(ids)>max_length: raise ValueError('overlength conversation; curate explicitly')
    if not ids or not any(x!=-100 for x in labels[1:]): raise ValueError('no supervised target')
    return {'input_ids':ids,'labels':labels}
 
def collate(rows,pad_id):
    if not rows: raise ValueError('empty batch')
    width=max(len(x['input_ids']) for x in rows)
    ids=torch.full((len(rows),width),pad_id,dtype=torch.long)
    labels=torch.full_like(ids,-100); attention=torch.zeros_like(ids)
    for i,row in enumerate(rows):
        n=len(row['input_ids'])
        if len(row['labels'])!=n: raise ValueError('mask length')
        ids[i,:n]=torch.tensor(row['input_ids']); labels[i,:n]=torch.tensor(row['labels'])
        attention[i,:n]=1
    return ids,labels,attention
 
def sft_loss(logits,labels):
    targets=labels[:,1:]; n=int((targets!=-100).sum())
    if not n: raise ValueError('no assistant targets')
    if logits.shape[:2]!=labels.shape: raise ValueError('logit/label shape')
    return F.cross_entropy(logits[:,:-1].reshape(-1,logits.shape[-1]),targets.reshape(-1),
                           ignore_index=-100,reduction='sum')/n
 
def local_sft(model,train_rows,valid_rows,steps=50,lr=.01,pad_id=0):
    if steps<1 or lr<=0: raise ValueError('training budget')
    opt=torch.optim.AdamW(model.parameters(),lr=lr,weight_decay=0)
    x,y,_=collate(train_rows,pad_id)
    vx,vy,_=collate(valid_rows,pad_id)
    # Right padding cannot influence earlier valid positions under this decoder's causal mask.
    # A general left-padded/HF model must consume an appropriate attention mask too.
    with torch.no_grad(): initial=float(sft_loss(model(x),y))
    best=float('inf'); best_state=None; logs=[]
    for k in range(steps):
        model.train(); opt.zero_grad(set_to_none=True)
        loss=sft_loss(model(x),y)
        if not torch.isfinite(loss): raise FloatingPointError('SFT loss')
        loss.backward(); torch.nn.utils.clip_grad_norm_(model.parameters(),1.,error_if_nonfinite=True)
        opt.step(); model.eval()
        with torch.no_grad(): val=float(sft_loss(model(vx),vy))
        if val<best: best=val; best_state=copy.deepcopy(model.state_dict())
        logs.append({'update':k+1,'train':float(loss.detach()),'validation':val})
    model.load_state_dict(best_state)
    return {'initial_train':initial,'best_validation':best,'history':logs}
 
if __name__=='__main__':
    torch.set_num_threads(1); torch.manual_seed(12)
    rows=[encode_segments([([1,2,3],False),([4],True)],0,12,8),
          encode_segments([([1,5],False),([6,7],True)],0,12,8)]
    x,labels,mask=collate(rows,0)
    assert int((labels[:,1:]!=-100).sum())==5
    assert labels[0,4]==0 and labels[1,4]==0  # real EOS remains scored
    z=torch.zeros(2,5,8,requires_grad=True); loss=sft_loss(z,labels); loss.backward()
    torch.testing.assert_close(loss,torch.tensor(8.).log())
    assert torch.count_nonzero(z.grad[0,:2])==0
    assert torch.count_nonzero(z.grad[0,2:4])>0
    for bad in [([([1,2],False)],12),([([1,2,3],False),([4],True)],3)]:
        try: encode_segments(bad[0],0,bad[1],8)
        except ValueError: pass
        else: raise AssertionError('invalid supervision accepted')
    model=Decoder(Config(vocab=8,width=8,layers=1,heads=2,kv_heads=1,hidden=16,context=12))
    # Validation uses separate examples with the same synthetic rule, not a natural-language benchmark.
    valid=[encode_segments([([1,2,2,3],False),([4],True)],0,12,8)]
    result=local_sft(model,rows,valid)
    assert result['history'][-1]['train']<result['initial_train']
    model.zero_grad(); sft_loss(model(x),labels).backward()
    assert model.embed.weight.grad[1].abs().sum()>0  # conditioning still receives gradient
    print('mask/EOS/shift/truncation checks and 50-update local SFT passed',
          result['initial_train'],result['best_validation'])

A complete local-data adaptation substitutes reviewed tokenized examples and a compatible trusted local base checkpoint, verifies its architecture/tokenizer/template hashes, and keeps entity/source-grouped train/validation/test splits. local_sft trains full parameters and selects by held-out loss; add the full restart state from Checkpointing for an actual long run. The bundled50-step run is an implementation test, not evidence that50 steps or1K examples suffice for a user task.

Chat templates are a versioned protocol

ChatML commonly uses role delimiters such as <|im_start|>/<|im_end|>. Llama3 Instruct uses header delimiters and an end-of-turn token; end-of-turn and end-of-text are distinct semantics. Early Mistral Instruct variants use [INST]...[/INST] with version-specific system-role handling. Do not declare all Zephyr/OpenAI/Mistral variants identical or memorize a family name instead of reading the exact tokenizer's template.

Prefer the saved tokenizer's apply_chat_template. A complete training conversation normally omits an extra generation prompt; inference with no final assistant message adds the appropriate assistant prefix if the template supports it. continue_final_message represents a different prefill operation. If rendering text first and tokenizing later, avoid duplicate BOS/EOS by respecting add_special_tokens=False where required. A base tokenizer may have no chat template; choose/add/train/save one deliberately rather than assume messages conversion supplies it.

Subword tokenization of a product name is normal, not proof that splitting into more than two pieces caused hallucination. Adding tokens changes embedding/output rows and initialization behavior; train the new rows, save the resized model/tokenizer together and evaluate. Prefer retrieval/copying constraints for exact changing product facts where appropriate. No invented USD12K client story or USD400 fix is evidence.

Packing, boundaries and effective budgets

Packing saves padding only to the extent padding was present. Dynamic batching/bucketing is a useful baseline; a60-token median does not imply97% wasted compute for every collator. Concatenating EOS-separated conversations still allows cross-example attention under a normal causal mask. If examples must be independent, use block-diagonal or variable-length attention with document boundaries and consistent position IDs. Test by changing example1 and checking example2 outputs under the isolated mask.

TRL behavior depends on release, packing strategy and attention backend. It is false that all versions since0.9/0.11 automatically guarantee isolation merely when packing=True. FlashAttention's varlen cumulative lengths encode boundaries for a compatible implementation; they are not magic metadata the model always consumes. The original2022 packing citation cannot support a claimed later FA2/MT-Bench gain without the actual experiment.

With50K examples averaging200 total tokens, three epochs process30M input tokens before packing/truncation. Supervised response count may be much smaller. For7B dense full-parameter training,6ND gives a rough1.26e18-FLOP proxy for30M tokens, subject to the usual attention/output/recomputation limitations. LoRA still performs frozen-base forward and input-gradient work; it does not eliminate almost all computation. Estimate peak memory and measure pilot throughput separately. LR2e-5 is a candidate, not a guaranteed30× safety margin, and there is no universal three-epoch cap.

Pinned TRL recipe using existing local artifacts

This is a resource-dependent reference, unexecuted here. Source contract: TRL0.23.1, Transformers4.57.1; the CPU environment does not have this integration installed. No install commands or network model IDs are used. Use a fully compatible existing local model/tokenizer with a validated template, and existing JSONL files containing conversational prompt and completion fields. The normalizer in Instruction Datasets produces the user/assistant content; split before tokenization.

# Resource reference: local-only TRL integration; not executed in this audit.
def train_local_trl(model_dir,train_jsonl,valid_jsonl,out_dir):
    import os,json
    from pathlib import Path
    os.environ['HF_HUB_OFFLINE']='1'; os.environ['HF_DATASETS_OFFLINE']='1'
    os.environ['TRANSFORMERS_OFFLINE']='1'
    import torch
    from datasets import Dataset
    from transformers import AutoModelForCausalLM,AutoTokenizer
    from trl import SFTConfig,SFTTrainer
    root=Path(model_dir)
    if not root.is_dir(): raise ValueError('existing local model required')
    tok=AutoTokenizer.from_pretrained(str(root),local_files_only=True,trust_remote_code=False)
    if not tok.chat_template or tok.eos_token_id is None: raise ValueError('template/EOS contract')
    if tok.pad_token_id is None: tok.pad_token=tok.eos_token
    def read_rows(path):
        rows=[json.loads(line) for line in Path(path).read_text().splitlines() if line.strip()]
        if not rows: raise ValueError('empty data')
        for row in rows:
            if not row['prompt'] or not row['completion']: raise ValueError('empty prompt/completion')
            if any(m['role']!='assistant' for m in row['completion']): raise ValueError('completion role')
            ids=tok.apply_chat_template(row['prompt']+row['completion'],tokenize=True,
                                        add_generation_prompt=False)
            if len(ids)>1024: raise ValueError('curate overlength rows before training')
        return Dataset.from_list(rows)
    tr,va=read_rows(train_jsonl),read_rows(valid_jsonl)
    model=AutoModelForCausalLM.from_pretrained(str(root),local_files_only=True,
                                              trust_remote_code=False,torch_dtype=torch.float32)
    cfg=SFTConfig(output_dir=str(out_dir),num_train_epochs=1,learning_rate=2e-5,
        per_device_train_batch_size=1,gradient_accumulation_steps=1,
        per_device_eval_batch_size=1,max_length=1024,packing=False,
        completion_only_loss=True,assistant_only_loss=False,report_to='none',
        bf16=False,fp16=False,gradient_checkpointing=False,eval_strategy='epoch',
        save_strategy='epoch',load_best_model_at_end=True,metric_for_best_model='eval_loss',
        greater_is_better=False,seed=42)
    trainer=SFTTrainer(model=model,args=cfg,train_dataset=tr,eval_dataset=va,processing_class=tok)
    probe=next(iter(trainer.get_train_dataloader()))
    if not (probe['labels'][:,1:]!=-100).any(): raise ValueError('no scored completion')
    # Inspect decoded inputs and every scored target span before opting into a long run.
    for ids,lab in zip(probe['input_ids'],probe['labels']):
        print(tok.decode(ids.tolist()),tok.decode(lab[lab!=-100].tolist()))
    before=trainer.evaluate(); trainer.train(); after=trainer.evaluate()
    trainer.save_model(str(out_dir)); tok.save_pretrained(str(out_dir))
    return {'before':before,'after':after,'execution':'local full recipe; evaluate held-out tasks separately'}

In this pinned version max_length and processing_class replace the original mixed-version max_seq_length/tokenizer recipe. A messages dataset does not automatically imply assistant-only loss: assistant_only_loss=True requires an appropriate conversational/template mask path, including generation tags where the tokenizer API needs them. This reference uses explicit prompt-completion supervision and disables packing to keep the first audit tractable. Inspect all prepared rows/masks before production, not only the illustrative first probe. A final deployment needs same-template bounded generation and regression evaluation, not just trainer.evaluate().

Failure modes and realistic interventions

Refusal collapse: audit benign helpfulness, unsafe-request handling and over-refusal separately, with labeled examples and source/task sampling weights. Short refusals are not automatically overweight under token averaging. Do not erase all refusals or impose a universal5% cap to make an assistant “helpful.” Correct mislabeled or out-of-scope demonstrations while preserving intended safety behavior.

Format drift: measure response prefixes, length and requested format on held-out prompts. Alpaca's original generator was text-davinci-003, not GPT-4; an opener like “Sure” is not a model-identity detector. Check labels/template/stop tokens before blaming epoch count. Conciseness is task-dependent; mixing contradictory response styles without conditioning can confuse the target policy.

Catastrophic forgetting: evaluate base and adapted models on both intended domain and retained capabilities, using matched protocols and paired uncertainty. Lower LR, early selection, regularization, replay data or adapters may help. LoRA freezes base parameters but the effective adapted function can still forget; it is not immune to regression. Disabling an adapter restores the base path only under the actual implementation/state contract.

No fixed initial2–4nat or final0.8nat band identifies a valid SFT run. Deterministic special tokens tend to be easy and can lower, not automatically raise, the mean. Train loss falling while validation/behavior degrades can indicate overfit; a plateau can indicate underfit or optimization issues. Compare relevant diagnostics before adding epochs.

Research landscape without universal prescriptions

**Alpaca:**52K generated demonstrations and historical costs belong to its2023 report and license/usage limits, not a current USD600 general budget. **LIMA:**65B LLaMA,1000 carefully curated examples and no RL/preference training in the reported setup. Its abstract says equivalent or preferred to GPT-4 in43% of comparisons and65% versus DaVinci003; the original lesson reversed the comparator. This is not proof that1000 examples universally beat52K or that all examples were freshly hand-written.

AlpaGasus studies filtering instruction data; it supports considering quality, not a global optimum dataset size. Tulu3 releases SFT, DPO and verifiable-reward post-training stages; final-model gains cannot be attributed entirely to persona-SFT. Its roughly939K SFT mixture already contradicts the claim that everyone stopped using large mixtures. Persona conditioning varies prompt voices/contexts; do not invent a10× diversity theorem or say speakers always matter more than tasks.

WildChat, UltraChat, OpenHermes and Magpie are distinct data products: real-user interaction logs, synthetic multi-turn dialogues, a curated mixture, and template-prefix-based synthetic generation. Version/subset, provenance and terms matter. Magpie avoids hand-written seed prompts, not inference cost. Verifiable math/code can support filtering, but do not train on held-out GSM8K/HumanEval test items and then report them as independent evaluation. No claim that most2025 production systems are SFT-only is established here.

Worked recall and experiment answers

1. What distinguishes assistant-only SFT? A chosen supervised label mask and demonstrations; SFT in general need not exclude all prompt loss. Prompts still carry conditioning gradients.
2. Loss starts at8? Inspect actual tokens/masks, tokenizer, domain and checkpoint; the scalar alone does not prove a missing mask.
3. Packing speedup? Derive saved padding from length distribution, backend and boundary handling; no fixed5–20× claim.
4. Llama3 stop? Use that exact tokenizer/generation config's end-of-turn/end-of-text semantics, not a guessed universal EOS.
5. Everything refuses? Audit labels and benign/unsafe slices, rebalance by intended behavior and verify both refusal quality and over-refusal; do not indiscriminately remove safety examples.
6. **LR from6e-4 base?**2e-5 is a30× lower candidate, not a proven optimum. Sweep a bounded validation-selected budget and monitor regressions; full/adaptor LR ranges are hypotheses.
7. Alpaca template echo after5epochs? Test scored spans, served template, stop config and examples; select earlier checkpoints or repair data based on evidence, not a universal epoch rule.
8. Equal loss, different templates? Different conditional distributions/targets make losses incomparable; serving with a mismatched protocol can worsen outputs despite equal training loss.
9. What does LIMA imply? Its setting supports the superficial-alignment hypothesis, not the impossibility of learning facts/capabilities or guaranteed success on a100M model.

For the100/1K/10K subset stretch: split by task/entity before choosing subsets; compare random and task-stratified curated subsets under declared token/update budgets and repeated seeds. Evaluate formatting, correctness, stopping, safety and retained skills. Five prompts are a debugging probe, not persuasive generalization evidence. Predict uncertainty and saturation, not invented scores.

Sources and execution boundary

TRL0.23.1 SFTConfig, trainer, Transformers4.57.1 templates, Alpaca, LIMA, instruction loss, Tulu3.

Executed: exact synthetic CPU masked-SFT code and independent gradient/boundary controls. Unexecuted: TRL integration, real Alpaca/domain training, pretrained model loading, GPU timings and quality claims. No dependencies, datasets or model weights were installed/downloaded. Optional reading and next steps are listed below. Review tomorrow: mark every scored target and explain how an ignored prompt embedding still receives gradient.

Inherited optional resources and navigation

These pre-read links are optional further reading; their timestamps, video claims and optional pacing are not accepted technical evidence.

• 1hr Talk Intro to Large Language Models
• Fine-tuning Large Language Models (LLMs) | w/ Example Code
• Fine Tuning LLM Models – Generative AI Course
• TRL SFTTrainer documentation
• Alpaca: A Strong, Replicable Instruction-Following Model

Previous: ← DL S057 · Next: DL S059 →