Instruction datasets: provenance, normalization and mixtures

Specify the job before selecting the data

Deep Learning & LLMs From Scratch · Session 59 / 80 · M10 · Fine-tuning + alignment.

Goal: identify instruction-data artifacts and their terms, normalize conversations without losing roles, audit quality/provenance, and implement a reproducible mixture. Data strongly affects the learned behavior, but “the same model always differs by15MMLU points” is not an evidence-backed constant. Architecture, optimization, formatting and evaluation remain important.

Prerequisites: revisit S058: SFT loss, response masks and chat templates. The inherited prerequisite named S022 but called it cross-entropy: S022 is Weight Initialization; for the intended loss background use DL005: probability and cross-entropy. Row counts and supervised-token counts differ, as the mixture calculation below shows.

Write a data specification first: target users/tasks/languages; output format and length; appropriate helpfulness/refusal behavior; freshness and privacy constraints; split unit; token budget; and failure criteria. Public, human-written and model-generated data have different benefits, not a fixed quality ranking. Human authorship does not automatically mean you own all rights, and synthetic generation does not automatically mean consistent correctness or a hard ceiling at the teacher's test score.

Dataset identity and licensing: name the artifact

The following are dated source/card facts, not commercial legal clearance. Read the actual license, source-component terms, model terms, collection consent/privacy constraints and intended-use restrictions for your jurisdiction/use. Research is not a blanket exemption. A cleaned copy does not erase original obligations, and data restrictions do not have one universal automatic propagation rule for every trained model.

Alpaca can establish a historical baseline; Dolly offers human-authored task demonstrations; OA offers rated multilingual conversation trees; FLAN offers task/prompt variation; synthetic explanation/evolution data can broaden difficult tasks. “Dolly is always concise,” “FLAN kills chat feel” and “more complex instructions add reasoning capacity” need actual distribution/task measurements, not dataset-name stereotypes.

What LIMA does and does not establish

LIMA's reported setting is a65B pretrained LLaMA fine-tuned on1000 carefully curated prompts/responses without RL or preference modeling. Its abstract reports equivalent-or-preferred responses to GPT-4 in43% of cases, Bard58%, DaVinci00365%;43% was not the DaVinci comparison. Its superficial-alignment hypothesis emphasizes the role of pretraining and response formats; it is not a proof that SFT cannot teach facts or that additional high-quality data always dilutes behavior.

Curation includes diversity, helpful responses and coherent conventions. No universal1B/3B/30B parameter threshold separates “volume” from “quality,” and multi-turn behavior requires suitable evidence, not merely a larger row count. A small curated subset can miss domain/task/language coverage; a larger diverse set can help. Compare subset sizes under a declared token/update budget and track both retained skills and intended behaviors.

Mixtures: examples, tokens and gradients differ

If an epoch concatenates source i with N_i examples, example exposure is proportional to N_i. Under token-averaged loss, longer supervised responses can contribute more terms. If source i is sampled with probability p_i and mean supervised length l_i, its expected share of supervised tokens is

This is not literally its fraction of gradient norm: gradient directions/magnitudes, clipping and per-batch normalization differ. Equal-example objectives weight differently again. Fifteen thousand400-token responses contain6M response tokens;50K60-token responses contain3M. Counting rows alone misses the reversal.

Temperature mixing sets

T=1 is size-proportional; T→∞ approaches source-uniform. T=2 gives square-root sizes. It compresses size ratios without proving optimal quality. Doubling any source multiplies its unnormalized weight by the same 2^(1/T) factor; the normalized change depends on its previous share, not a general “small sources change more” law.

For52K,15K and1M examples, concatenation gives the last source about93.72% of examples. Square-root mixing gives approximately16.89%,9.07%,74.05%, not the original15/8/77. For five sources52K/15K/161K/250K/1M, compute actual normalized weights rather than copying the inaccurate printed vector. Raw concatenation is acceptable if that exposure matches the intended objective; neither temperature nor hand-set weights is universally superior.

Fixed-probability interleaving is not size-proportional merely because both were called “proportional sampling.” Stopping at the first exhausted source versus cycling until all are exhausted changes epoch size/repetition. Our sampler below chooses an explicit finite number of draws with replacement and reports exposure. For S draws, expected appearances per original example in source i are S*p_i/N_i; a tiny gold set can be revisited heavily.

Measure the supervised response stream, not a standalone response string. The following CPU program measures each source's accepted example count, total and median supervised lengths. Its callable tokenizer is an authored regex fixture, not any model's tokenizer. Counts below are actual fixture measurements, not estimates for Alpaca, Dolly or a deployed model. Supply the same rendered prompt/response text, tokenizer revision and special-token policy as your training pipeline before drawing real mixture conclusions.

Tokenizing prompt and response separately can change their boundary tokens. We tokenize their concatenation once, inspect character offsets, and reject a token straddling the boundary instead of assigning it an invented partial label. A real tokenizer must supply trustworthy offsets in the same Python-character coordinate system; unsupported/byte-offset tokenizers need a reviewed adapter. This plain-text, single-final-response example does not implement chat templates, assistant masks for multiple turns, or packing. A templated trainer must instead verify its exact template and mask. BOS is always masked; EOS is deliberately a response target, even when PAD shares its ID. No padding is added. After explicit right truncation, count labels[1:], the targets available after the causal one-position shift; reject a response with no surviving content token (EOS alone is insufficient).

# CPU-only teaching tokenizer and per-source measurement. No external packages.
import re
from statistics import median
 
class FixtureTokenizer:
    bos_token_id, eos_token_id, pad_token_id = 1, 2, 2
    def __init__(self):
        self.vocab = {}
    def __call__(self, text, *, add_special_tokens=False,
                 return_offsets_mapping=True):
        if add_special_tokens or not return_offsets_mapping:
            raise ValueError('fixture requires explicit special-token policy and offsets')
        pieces = list(re.finditer(r'\w+|[^\w\s]', text, flags=re.UNICODE))
        ids = [self.vocab.setdefault(m.group(), len(self.vocab) + 3) for m in pieces]
        return {'input_ids': ids, 'offset_mapping': [m.span() for m in pieces]}
 
def response_record(tok, prompt, response, max_length):
    if not isinstance(prompt, str) or not isinstance(response, str) or not response.strip():
        raise ValueError('nonempty response and string prompt required')
    if type(max_length) is not int or max_length < 2:
        raise ValueError('max_length must be an integer at least two')
    if tok.bos_token_id is None or tok.eos_token_id is None:
        raise ValueError('explicit BOS and EOS required')
    text = prompt + response
    boundary = len(prompt)
    encoded = tok(text, add_special_tokens=False, return_offsets_mapping=True)
    ids, offsets = encoded['input_ids'], encoded['offset_mapping']
    if len(ids) != len(offsets):
        raise ValueError('offset alignment')
    content_mask = []
    previous_end = 0
    for start, end in offsets:
        if not (previous_end <= start < end <= len(text)):
            raise ValueError('non-overlapping character offsets required')
        if start < boundary < end:
            raise ValueError('token crosses prompt/response boundary')
        content_mask.append(start >= boundary)
        previous_end = end
    if not any(content_mask):
        raise ValueError('no response content token')
    full_ids = [tok.bos_token_id] + list(ids) + [tok.eos_token_id]
    mask = [False] + content_mask + [True]
    labels = [token if selected else -100 for token, selected in zip(full_ids, mask)]
    content_after_cut = ([False] + content_mask + [False])[:max_length]
    if not any(content_after_cut[1:]):
        raise ValueError('truncation removes all response content')
    kept = full_ids[:max_length]
    labels = labels[:max_length]
    return {'input_ids': kept, 'attention_mask': [1] * len(kept), 'labels': labels,
            'supervised_length': sum(label != -100 for label in labels[1:]),
            'full_supervised_length': sum(mask[1:]),
            'truncated': len(full_ids) > max_length,
            'eos_retained': len(full_ids) <= max_length}
 
def measure_sources(sources, tok, max_length):
    report = {}
    for source, examples in sources.items():
        lengths, full_lengths, rejects = [], [], []
        truncated = 0
        for index, example in enumerate(examples):
            try:
                row = response_record(tok, example['prompt'], example['response'], max_length)
            except ValueError as exc:
                rejects.append({'index': index, 'reason': str(exc)})
                continue
            lengths.append(row['supervised_length'])
            full_lengths.append(row['full_supervised_length'])
            truncated += row['truncated']
        report[source] = {'n_input': len(examples), 'n_accepted': len(lengths),
                          'n_rejected': len(rejects), 'truncated': truncated,
                          'total': sum(lengths),
                          'median': median(lengths) if lengths else None,
                          'mean': sum(lengths) / len(lengths) if lengths else None,
                          'full_total_for_accepted': sum(full_lengths), 'rejects': rejects}
    return report
 
fixture_sources = {
    'brief': [{'prompt': 'Q: ', 'response': 'Yes.'},
              {'prompt': 'Q: ', 'response': 'No'}],
    'long': [{'prompt': 'Q: ', 'response': 'One two three four.'}],
    'empty': []}
fixture_report = measure_sources(fixture_sources, FixtureTokenizer(), max_length=7)
assert (fixture_report['brief']['total'], fixture_report['brief']['median']) == (5, 2.5)
assert (fixture_report['long']['total'], fixture_report['long']['median']) == (4, 4)
assert fixture_report['long']['truncated'] == 1
assert fixture_report['long']['full_total_for_accepted'] == 6
assert fixture_report['empty']['median'] is None
print(fixture_report)

Here brief has supervised lengths 3 and 2 (content plus EOS), whereas long keeps four content tokens and loses the final punctuation and EOS at the seven-token cap. Equal source sampling therefore gives expected supervised-token shares 2.5/(2.5+4) and 4/(2.5+4), approximately 38.46% and 61.54%, not equal shares. Use the mean, not the median, in the mixture formula; the median is a distribution diagnostic. Totals describe the accepted source census, not a with-replacement sampled epoch. Report rejections/truncation before selecting probabilities, and recompute from the actual sampled, masked stream if sampling changes it. Do not infer gradient norm or output style from either statistic. Practice: prompt='Hel', response='lo' must be rejected by this fixture because the full token is Hello; max_length=3 for Q: /Yes. must be rejected because only BOS and prompt tokens survive. Neither failure is repaired by counting response words separately.

The selected-target convention follows the ignored-label contract in PyTorch 2.9.0 CrossEntropyLoss; the causal shift and offset policy are explicit choices of this authored measurement algorithm, not behavior attributed to a model tokenizer.

Complete local normalization and audit program

Save as instruction_data.py. It reads ordinary Python records supplied locally; no datasets library or network access is required. Normalization preserves the original record/provenance outside the normalized messages. It does not silently strip meaningful text merely because it contains “Assistant:” or a safety-related opener.

import json,hashlib,math,random,re
from collections import Counter
 
ROLES={'system','user','assistant'}
def normalize(record,kind):
    if kind=='alpaca':
        prompt=record['instruction'].strip()
        if record.get('input','').strip(): prompt+='\n\n'+record['input'].strip()
        messages=[{'role':'user','content':prompt},{'role':'assistant','content':record['output']}]
    elif kind=='dolly':
        prompt=record['instruction'].strip()
        if record.get('context','').strip(): prompt+='\n\nContext:\n'+record['context'].strip()
        messages=[{'role':'user','content':prompt},{'role':'assistant','content':record['response']}]
    elif kind=='messages': messages=[dict(m) for m in record['messages']]
    else: raise ValueError('explicit schema required')
    if not messages: raise ValueError('empty conversation')
    expected='user'
    for i,m in enumerate(messages):
        if m.get('role') not in ROLES or not isinstance(m.get('content'),str) or not m['content'].strip():
            raise ValueError('role/content')
        if m['role']=='system':
            if i!=0: raise ValueError('system only at beginning in this format')
            continue
        if m['role']!=expected: raise ValueError('turn alternation')
        expected='assistant' if expected=='user' else 'user'
    if messages[-1]['role']!='assistant': raise ValueError('no terminal assistant demonstration')
    return messages
 
def oa_paths(records):
    by_id={r['message_id']:r for r in records}
    if len(by_id)!=len(records): raise ValueError('duplicate message ID')
    children={k:[] for k in by_id}; roots=[]
    for k,r in by_id.items():
        p=r.get('parent_id')
        if p is None: roots.append(k)
        elif p not in by_id: raise ValueError('missing parent')
        else: children[p].append(k)
    result=[]; visited=set()
    def walk(k,path,active,root):
        if k in active: raise ValueError('tree cycle')
        visited.add(k); r=by_id[k]
        if r.get('deleted') or r.get('review_result') is False: return
        role={'prompter':'user','assistant':'assistant'}.get(r['role'])
        if role is None: raise ValueError('OA role')
        path=path+[{'role':role,'content':r['text']}]
        if role=='assistant':
            result.append({'group':root,'messages':normalize({'messages':path},'messages')})
        for child in sorted(children[k]): walk(child,path,active|{k},root)
    for root in sorted(roots): walk(root,[],set(),root)
    # Inspect every parent chain, including filtered branches, for cycles.
    for k in by_id:
        seen=set(); j=k
        while j is not None:
            if j in seen: raise ValueError('tree cycle')
            seen.add(j); j=by_id[j].get('parent_id')
    return result
 
def canonical(messages):
    # Unicode whitespace normalization is declared; preserve role and case/content otherwise.
    return json.dumps([{'role':m['role'],'content':' '.join(m['content'].split())} for m in messages],
                      ensure_ascii=False,sort_keys=True,separators=(',',':'))
 
def audit(rows):
    seen={}; unique=[]; rejected=[]; openers=Counter(); flags=[]
    for row in rows:
        for field in ('source','source_id','group','license','messages'):
            if field not in row: raise ValueError('missing provenance '+field)
        msg=normalize(row,'messages'); digest=hashlib.sha256(canonical(msg).encode()).hexdigest()
        if digest in seen:
            rejected.append({'source_id':row['source_id'],'duplicate_of':seen[digest],
                             'group':row['group'],'sha256':digest}); continue
        seen[digest]=row['source_id']; result={**row,'messages':msg,'sha256':digest}; unique.append(result)
        response=msg[-1]['content']; openers[' '.join(response.split()[:3])]+=1
        reasons=[]
        if any(s in response for s in ['### Instruction:','Human:','Assistant:','[INST]','<|im_start|>']):
            reasons.append('template-like literal; human review, not automatic deletion')
        if re.search(r'\b[\w.+-]+@[\w.-]+\.[A-Za-z]{2,}\b',response):
            reasons.append('possible email; privacy review, incomplete detector')
        if reasons: flags.append({'source_id':row['source_id'],'reasons':reasons})
    return {'records':unique,'duplicates':rejected,'flags':flags,'openers':dict(openers)}
 
def group_split(rows,seed='fixed',valid_fraction=.2):
    if not 0<valid_fraction<1: raise ValueError('split fraction')
    # Caller must assign connected provenance/duplicate groups BEFORE this split.
    train=[]; valid=[]
    for row in rows:
        value=int(hashlib.sha256((seed+'|'+row['group']).encode()).hexdigest()[:16],16)/2**64
        (valid if value<valid_fraction else train).append(row)
    assert not {r['group'] for r in train}&{r['group'] for r in valid}
    return train,valid
 
def mixture(sizes,temperature=2):
    if not sizes or any(not math.isfinite(n) or n<=0 for n in sizes) or not math.isfinite(temperature) or temperature<=0:
        raise ValueError('mixture parameters')
    logn=[math.log(n) for n in sizes]; top=max(logn)
    # Subtract before dividing: tiny positive temperatures must not produce inf-inf.
    w=[math.exp((x-top)/temperature) for x in logn]; return [x/sum(w) for x in w]
 
def draw_sources(sources,probabilities,draws,seed=7):
    if not sources or any(not s for s in sources) or len(sources)!=len(probabilities):
        raise ValueError('sources')
    if draws<0 or any(p<0 or not math.isfinite(p) for p in probabilities) or not math.isclose(sum(probabilities),1.):
        raise ValueError('probabilities/budget')
    rng=random.Random(seed); out=[]
    for _ in range(draws):
        i=rng.choices(range(len(sources)),weights=probabilities,k=1)[0]
        out.append((i,rng.choice(sources[i])))
    return out
 
if __name__=='__main__':
    msg=normalize({'instruction':'Add','input':'2 and 3','output':'5'},'alpaca')
    rows=[{'source':'local','source_id':str(i),'group':'same-tree','license':'local-test',
           'messages':msg} for i in range(2)]
    report=audit(rows); assert len(report['records'])==1 and len(report['duplicates'])==1
    tr,va=group_split(report['records']); assert len(tr)+len(va)==1
    tree=[{'message_id':'r','parent_id':None,'role':'prompter','text':'Hi'},
          {'message_id':'a','parent_id':'r','role':'assistant','text':'Hello'},
          {'message_id':'b','parent_id':'r','role':'assistant','text':'Welcome'}]
    paths=oa_paths(tree); assert len(paths)==2 and {p['group'] for p in paths}=={'r'}
    try: oa_paths([{'message_id':'r','parent_id':'r','role':'prompter','text':'x'}])
    except ValueError: pass
    else: raise AssertionError('cycle accepted')
    p=mixture([52000,15000,1000000]); assert math.isclose(sum(p),1.)
    a=draw_sources([[1,2],[3]],[.5,.5],20); assert a==draw_sources([[1,2],[3]],[.5,.5],20)
    assert mixture([1,100],1)==[1/101,100/101] or math.isclose(mixture([1,100],1)[0],1/101)
    print('normalization/tree/dedup/split/sampler checks passed; sqrt mixture:',p)

The mixture helper requires finite positive source sizes and a finite positive temperature. Subtracting the largest log-size before dividing avoids inf - inf at very small temperatures; at machine precision, smaller weights may underflow to zero, approximating largest-source-only sampling. That numerical limit is not evidence of a good data policy. The source sampler expects integer draw counts and nonempty local source lists; schema and provenance checks remain separate.

For a full local JSONL pipeline, load each source with json.loads per line, choose its explicit normalizer, retain source revision/license/generator/collection date and source IDs, reconstruct OA branches, apply reviewed quality/privacy rules, establish connected provenance/duplicate groups, split, then apply within-split dedup/audit and the finite sampler. Export JSONL with messages and provenance; for the SFT prompt-completion recipe, split the final assistant turn as completion and preceding turns as prompt. Save rejected IDs/reasons and sampling seed/budget/counts. Do not discard upstream lineages when schemas are normalized.

The provided exact dedup catches normalized identical conversations; it does not discover all near-duplicates or automatically union duplicate groups. If two sources contain the same conversation, either merge their groups before split or enforce a predeclared cross-split exclusion with a report. MinHash/LSH can generate near-duplicate candidates for reviewed Jaccard/shingle thresholds; they are not paraphrase guarantees. A single file's random row split can put branches of the same OA tree on both sides.

Quality audits without destructive heuristics

Template remnants: search role delimiters, headers and escaped artifacts, then inspect context. A valid answer about chat templates may intentionally contain those strings. Flagging is not permission to drop all matches. Preserve original and normalized text with transformation reasons.

Preambles: count first words and response lengths by source/task/language. “Sure,” “Certainly” or “As an AI” is a style indicator, not proof of GPT-4 authorship or an infection threshold. Compare rates in the sampled token stream, not only raw source tables. Removing a polite opener may be appropriate for an exact-output task but destructive for another domain.

Role confusion: validate roles/order/tool-call schema and inspect labeled samples. User turns can be long and assistant answers short; length is a triage signal, not a label correction rule. Do not infer a role from fewer than200tokens. For tools/multimodal messages extend the explicit schema rather than silently flattening to ordinary text.

Safety and privacy: distinguish justified refusals, over-refusal, harmful demonstrations, quoted harmful content and benign explanations. Do not delete all refusals or cap them at5% by habit. Review consent, personal data, permissions, retention/deletion lineage and intended use. The toy email detector is deliberately incomplete, not a privacy certification. Repetitive synthetic answers can be fluent but factually wrong; inspect accepted and rejected samples for bias.

Synthetic pipelines and their failure modes

Self-Instruct: bootstrap instructions/inputs/outputs, filter invalid and similar examples, then fine-tune. The original method used GPT-3; Alpaca adapted the idea with text-davinci-003 and its historical generation recipe. Do not conflate both with one175seed/GPT-3.5/52K pipeline. Track generator revision, prompt, sampling, seed, cost and license. No paid generation was executed here.

Evol-Instruct: rewrite seed instructions for depth, constraints, reasoning or breadth, then generate/validate responses. Harder wording is not automatically a better task: it may introduce contradictions, unverifiable premises or unnatural verbosity. For the original recurrence exercise, a0=1,a1=2,a_n=3a_(n-1)-2a_(n-2) gives2^n; the alternate characteristic roots are (3±sqrt17)/2 and constants depend on initial conditions. A synthetic generator must preserve these constraints rather than merely sound difficult. No blanket accusation that a specific dataset always lifts benchmark tests is established.

Magpie: begin with an aligned model's user-role template prefix, sample a user query and then a response. The paper describes4M generated instructions/responses and a selected300K set in its comparison. “No seed prompts” does not mean no inference compute/cost or no teacher-distribution bias.

Failure controls: deduplicate and measure task/language/persona diversity; verify answerable problems; use exact answer/program tests only under a safe execution policy; retain sampled human audits; protect held-out benchmarks; detect privacy/unsafe content; version generator/prompts; compare errors and novelty. Filtering by another model's rubric can reproduce its biases. A student can sometimes improve through data selection, verification and optimization; it is not mathematically capped at the teacher's measured score.

Modern mixtures and a defensible 7B experiment

WildChat documents opt-in collection of1M conversations and over2.5M turns. Its paper describes request metadata and potentially toxic cases; do not invent a fractional “median3.2turns” or a universal13% toxicity rate. Consent to collection does not eliminate every privacy/reuse obligation.

Tulu3 SFT mixture has a useful provenance caution: the fetched card's structured metadata says939343 examples while prose says939344. Its 18 listed component counts sum to 889344, another 50000 below the prose total. These are discrepancies in the saved card, not an independently counted dataset; report all three totals rather than manufacture a reconciliation. It lists actual sources including OASST1, FLAN, No Robots, personas, WildChat, math/code, safety and domain sources—not the original invented component table. The collection notice says ODC-BY and explicitly warns that subset licenses differ and some portions are non-commercial. It is presented as a research artifact, not a commercially clean drop-in.

UltraChat versus UltraFeedback: demonstrations versus AI-feedback data used to construct preference/ranking training; chosen/rejected conversions depend on filtering/label policy. Zephyr's use of these artifacts is a recipe, not evidence that every open DPO paper uses them. OpenAssistant2, OpenHermes and other community mixtures require their own pinned cards, versions and component rights; their names are not proof of provenance. Persona-conditioned synthesis is a way to vary speakers/context and compare coverage; no10× improvement or sole-cause IFEval attribution is claimed.

For a7B candidate, start with an unchanged-base evaluation. Select locally available permitted datasets that match tasks, declare example/token budgets and trial weights, and compare concatenation, square-root mixing and a curated subset with repeated validation. A proposed180K mix or35/20/10/15/15/5 split is an experiment, not a standard. Decide chat template and added tokens jointly with the base; resize/train/save added embeddings. Use DL058's pinned local SFT recipe or DL061's local QLoRA reference, measure memory/throughput, select by target and regression metrics, and reserve a final test. Do not promise62%MMLU,24A100-hours or a guaranteed two-point regression cutoff.

Worked recall, quiz and design answers

• LIMA in one sentence: its65B/1000-example study supports effective curated alignment in that setting, not “SFT only teaches format” for every model.
• Why not concatenate? It fixes exposure by source size/length, which may or may not match the intended distribution. Compute shares before changing the sampler.
• Murky license? Any incompletely specified collection can be unclear; ShareGPT is not one universal license. No Robots is explicitly noncommercial in the fetched card, and Dolly/OA terms still need compliance.
• Preamble grep? Count common openers and inspect task context.20%,30% and35% are not causal refusal/style thresholds. Preserve appropriate safety demonstrations.
• When quality versus quantity “breaks”? Ask whether the curated data covers target tasks, languages and turns and whether the base supports them; no exact parameter cutoff settles it.
• **52K/15K/1M arithmetic:**93.72% raw example share for1M; square-root shares16.89/9.07/74.05%. Token lengths and loss normalization can change target contribution further.
• 500K noisy plus5K gold code examples: retain rights/source metadata and deduplicate by issue/repository/entity; reserve grouped holdouts; compare bounded noisy-first/gold-second and mixed runs. Naive equal-length concat gives gold about0.99% example share. At temperature2, relative weights are10:1, so gold≈9.09%; with550K draws it is revisited about10times/example. Curation and lower-LR second-stage training are candidate interventions, not a documented CodeLlama identity or guaranteed3–10× optimum.
• Commercial replacement? There is no universally clean substitute from a family name. Pick an exact permitted artifact and document all relevant terms instead of promising that human-authored subsets cleanse a mixture.

A200-word data spec should state inputs, rights, normalization, split/grouping, token/row weights, quality/privacy controls, generator lineage, bounded training and evaluation. The five-minute audit is a first sample inspection, not a certificate. Review tomorrow: compute both row and token shares and identify one legitimate response that a naive template grep would wrongly delete.

Sources and execution boundary

Alpaca, Dolly card, OASST1 card, No Robots card, LIMA, FLAN, Orca, Evol-Instruct, Self-Instruct, Tulu mixture, Magpie.

Executed: generated local records, schema/tree/cycle checks, exact dedup, grouped split and finite sampler/math controls. Cards/papers were retrieved as documentation; no datasets, model weights, generator API or7B training was used. This is not legal clearance. Optional reading and next steps are listed below.

Inherited optional resources and navigation

These pre-read links are optional further reading; their timestamps, video claims and optional pacing are not accepted technical evidence.

• Instruction finetuning and RLHF lecture (NYU CSCI 2590) — retained as an optional pointer, not an endorsement of the old “single best” praise or an independently verified employment attribution. The old 20-minute card and “first 40 minutes” instruction conflict; neither is a verified duration here.
• A Hackers' Guide to Language Models — the old ten-minute estimate and “advice you will not find in papers” praise are unverified, not requirements or evidence of unique content.
• Alpaca — a self-instruct dataset in 52K — a historical Self-Instruct adaptation, not proof that every later synthetic instruction dataset descends from it.
• LIMA — Less Is More for Alignment — the 65B/1,000-demonstration study did not use RL or preference modeling. Its reported response comparisons are setting-specific; “1,000 curated examples ≈ an RLHF-trained 65B model” is not a general equivalence. See What LIMA does and does not establish for the actual comparator percentages.

Previous: ← DL S058 · Next: DL S060 →