Where LLMs Came From, and What They Actually Are
A source-grounded history of language modelling, five core mechanisms and the boundaries between a predictor and a deployed assistant.
Published Updated
In this article
- Trace count models, learned representations, sequence models and attention
- Distinguish token IDs, embeddings, hidden states and model objectives
- Separate pretraining, supervised adaptation and preference optimization
- Evaluate claims about memory, tools, facts and understanding
Background & source details
Level: beginner
Bring with you
Sources checked: . A source check does not change the publication or update date.
Editorial review: · What review means
In this article · 21 sections
The one-paragraph version
A typical autoregressive language model estimates a distribution over the next token given a prefix. Repeated prediction can generate text, but a deployed assistant is more than that predictor: post-training changes behaviour, a conversation template supplies roles, and software may add retrieval, tools, persistent memory and safety checks. None of these makes fluent output proof of truth. Other language models use masked-token or denoising objectives, so “next-token prediction is the only LLM objective” is false.
For “The capital of France is…”, a base model may assign Paris high probability from learned parameters. A retrieval-enabled assistant may also consult a source. Ask which process actually occurred before saying either “it looked this up” or “there is no database.” This distinction—model computation versus the surrounding product—is the most useful beginner mental model.
Part 1 — How we got here
This is a selective history of mechanisms, not a chain in which every new method made its predecessor useless. Count models, recurrent models, retrieval and neural networks still coexist. Dates below identify particular publications, not claims that one paper invented all related ideas.
| Date and primary work | What changed | What did not follow automatically |
|---|---|---|
| 1948: Shannon, A Mathematical Theory of Communication | Successive probabilistic approximations to English make statistical structure visible. | This was information theory, not a modern trained chatbot or the sole origin of language research. |
| 1966: Weizenbaum, ELIZA | Keyword/decomposition/reassembly rules produced conversational responses; the DOCTOR script is the familiar example. | Plausible dialogue does not establish human-like understanding. ELIZA is not a transformer trained by backpropagation. |
| Count-based n-grams | Estimate a word from a bounded history using observed counts. | Unsmoothed estimates give zero to unseen events, but smoothing/backoff address that; it is wrong to say all n-grams necessarily assign zero. |
| 2003: Bengio and colleagues | Jointly learn word representations and a probabilistic neural language model, sharing statistical strength across similar contexts. | A learned vector is not a dictionary definition or a guarantee of semantic correctness. |
| 2013: Mikolov and colleagues | Efficient word-vector objectives make large-scale representation learning practical. | Some analogies emerge; all relations are not exact linear arithmetic. These vectors are not a complete chat system. |
| 2014: Sutskever, Vinyals and Le | An encoder/decoder LSTM maps a sequence to a fixed-dimensional representation and generates a target sequence. | The fixed-state route is a potential information bottleneck, not proof that the published system failed every long sentence. |
| 2014 preprint /2015 ICLR: Bahdanau, Cho and Bengio | A decoder learns alignment weights over source annotations rather than relying only on one final vector. | Weighted access is not perfect translation, nor an interpretable explanation of every decision. |
| 2017: Vaswani and colleagues | The Transformer removes recurrence/convolution from its core sequence architecture, using attention, feed-forward layers, positions, residuals and normalization. | It is not “only attention.” Teacher-forced training parallelizes positions; autoregressive generation still depends on prior generated tokens. |
| 2018–2020: GPT-family scaling, culminating in GPT-3 | Larger autoregressive models demonstrate tasks from instructions/examples in context; GPT-3 evaluates few-shot use without task-specific gradient updates. | Loss-scaling fits do not predict every downstream capability or make prompt examples equivalent to updating weights. |
| 2022: InstructGPT | Supervised demonstrations, preference comparisons, a reward model and reinforcement learning align outputs with the study's preferences. | Its 1.3B model being preferred to 175B GPT-3 on that evaluation is not a universal smaller-is-better theorem. |
| 2023 onward | Open-weight models, multimodal inputs, longer contexts, tool controllers and varied post-training methods expand the system design space. | “Latest,” “largest,” product memory and private architecture require dated evidence, not extrapolation from one model family. |
The practical arc is counts → learned representations → sequence models → selective access → parallelizable attention architectures → scaled training → task/preference adaptation. This is an organizing story, not proof that capability has one cause. The Bengio paper, Transformer paper, GPT-3 paper and InstructGPT paper provide explicit objectives and experimental boundaries.
Part 2 — The five ideas that actually matter
1. Tokens — text becomes indices
A tokenizer converts text into discrete IDs. Depending on the tokenizer, a token can represent a word, part of a word, punctuation, whitespace, a byte sequence or another unit. “Unbelievable splits into these three pieces” is an illustration until you run a named tokenizer. English words-per-token heuristics are not dependable for code, other languages or arbitrary text.
IDs select learned embedding rows; they are not ordinal meanings. ID 400 does not mean twice as much as ID 200. Tokenization affects length budgets, offsets and what patterns the model must learn. It can make character operations awkward, but it does not literally make spelling inaccessible: tokens can encode bytes and a model may learn spelling relationships. A character-counting error has multiple possible causes. Test a specified model/tokenizer rather than asserting that all models “cannot see letters.”
Worked trace: an invented vocabulary is ['the', 'cat', 'sat', '.']. Encoding “the cat sat .” as [0,1,2,3] says nothing about vector values. It only identifies rows. A real tokenizer must also define whitespace, unknown/byte fallback behaviour and decoding; this four-item demonstration is not such a tokenizer.
2. Embeddings — learned coordinates, not literal meaning
An embedding table maps each vocabulary ID to a vector. In a language model it is optimized along with other parameters. Shared learned features can support generalization to new combinations, unlike a bare count for each entire phrase. Similarity can also encode unwanted associations or conflate opposite concepts; proximity is not a truth relation.
Separate the input token embedding from the contextual hidden state after layers mix information. The same token “bank” starts from the same table row in a simple embedding lookup, but later states can differ between river-bank and financial-bank contexts. Separate both from a sentence embedding trained for retrieval; one vector type is not interchangeable with another.
Check: if you permute vocabulary IDs and permute the embedding/output rows and tokenizer consistently, the represented computation need not change. The arbitrary integer label is not the semantic content.
3. Attention — a precise weighted read
A query scores keys, the permitted scores are normalized, and the resulting weights mix values. Consider the pronoun in “The trophy did not fit in the suitcase because it was too big.” It motivates contextual reference, but attention is not a programmed rule saying “it means trophy.” Learned projections and the entire network determine the answer.
For a tiny read with weights .75 and .25 and value vectors [2,0] and [0,4], the output is [1.5,1]. Weights sum to one; the output is not a copy of either source. Changing values while keeping weights fixed changes the result. That is why an attention heatmap alone is not a full causal explanation.
Multi-head attention performs several learned projections/reads and combines them. Heads are not assigned permanent roles such as “grammar” and “subject matter” by the definition. A causal decoder masks future positions; an encoder can use bidirectional context. Dense attention has pairwise work quadratic in sequence length at fixed width, while tiled implementations can avoid materializing the whole score matrix. Some architectures replace or combine attention with recurrent/state-space mechanisms.
See the self-attention lesson for equations, masking and an executed numerical program, then multi-head attention for shapes and independent controls.
4. Next-token prediction — a useful objective, not the only one
For an autoregressive example, the model gives the correct next token probability p, and the loss is minus log p. If p=.5 the loss is about .693 nats; if p=.01 it is about 4.605. The gradient updates parameters to reduce average loss on training examples. At inference, greedy decoding selects a maximum-scoring token, while sampling draws according to a specified distribution; temperature changes that distribution, not factual verification.
Predicting varied text can reward learning syntax, facts, formatting and useful procedures. It does not logically force complete knowledge, grounded understanding or faithful reasoning. Training data, capacity, optimization, task structure and evaluation all matter. “Compression” is an intuition for predictive regularity, not an independent proof of cognition.
BERT uses masked-language-model pretraining and a next-sentence objective in the original paper. T5 studies a text-to-text framework and denoising pretraining. Preference learning, reinforcement learning and auxiliary objectives add further counterexamples to the claim that every training stage is next-token prediction on an ordinary text corpus.
5. Training stages — overlapping effects, distinct objectives
| Stage | What supplies the target/signal | What it can change |
|---|---|---|
| Pretraining | Large collections of text or multimodal examples and a predictive/denoising objective | Representations, learned associations, skills and biases—not guaranteed factual recall. |
| Supervised fine-tuning | Curated examples of desired outputs with a specified loss mask/template | Instructions, format, domain knowledge and behaviour; it is not restricted to style. |
| Preference/reward optimization | Pairwise choices, reward models, verifiable outcomes or other feedback | Relative response preferences under that feedback; it can improve or degrade capabilities and does not certify safety. |
The InstructGPT recipe is one important example: collect demonstrations, fit an SFT policy, collect comparisons, train a reward model and optimize the policy with PPO under regularization. Direct Preference Optimization (DPO) instead fits the policy directly from preference pairs using a classification loss, without a separately fitted reward model followed by a PPO loop. This changes the optimization procedure, not the need to examine whose preferences supplied the data. Not every deployed assistant uses the same pipeline. Human rankings are data with a population and rubric, not a universal definition of helpfulness.
For one capital-city question, pretraining might support the fact, SFT might teach an answer format and preference optimization might favour brevity. But those effects overlap: SFT can teach facts, preference learning can change accuracy and pretraining can already produce instruction-like responses. Compare behaviours using fixed prompts rather than assigning each stage one exclusive psychological role.
The mental model in one sentence
A language model learns statistical computations over representations; an assistant combines such a model with training choices, context, software and policies to perform tasks. Trace all of those before explaining an observed answer.
The five concepts are IDs and tokenization; learned/contextual representations; weighted information access; predictive objectives with real alternatives; and distinct but interacting adaptation stages. None is a truth switch.
What LLMs are not: model and product boundaries
| Misconception | Better question and explanation |
|---|---|
| “It always looks up facts.” | Did a retrieval/tool call occur? Parametric generation alone has no external lookup, but a product may attach one. |
| “It learns weights from every conversation.” | Ordinary inference usually keeps weights fixed. Context, saved product memory and later training-data policies are different mechanisms; closing a tab need not erase product storage. |
| “It has no persistence or tools by definition.” | A bare forward pass does not specify a product lifecycle. Software can retain state, schedule actions and expose tools; permissions must be enforced outside model prose. |
| “Every useful reasoning step must be visible in the answer.” | Visible intermediate text can help some tasks, but internal computation and private reasoning interfaces exist. A written rationale is not guaranteed faithful to the computation. |
| “Bigger always wins.” | Compare a specified task, data, budget, tools, training and uncertainty. A parameter count alone does not determine quality. |
| “Fluent means correct.” | Check consequential claims against evidence or executable tests. Confident style does not measure the probability of factual support. |
Current identity rule: a model name, architecture revision and product setting are separate facts. This article does not identify an unknown deployed assistant from its writing style or infer a proprietary model's parameter count. If the provider does not disclose a detail, say it is unknown. For a dated concrete example, the Kimi source study binds its claims to an official revision rather than to “the current best model.”
The argument worth knowing about
Bender, Gebru and colleagues' Stochastic Parrots critique raises risks from large language models, including bias, environmental/resource costs and readers attributing meaning to fluent form. Bender's 2021 author presentation makes the distinction between linguistic form and communicative meaning explicit. That is a critique to understand, not a licence to reduce every model output to a copied sentence.
The Sparks of Artificial General Intelligence authors report experiments with an early GPT-4 version and argue that its abilities motivate an early-AGI interpretation. Read their actual abstract and limitations, including the model/version and exploratory nature of the evaluation. These are the authors' interpretations; neither an impressive demonstration nor the phrase “autocomplete” settles a general theory of understanding.
A useful disagreement states an operational claim: which task, controls, counterexamples and evidence would change your view? Behavioural performance, causal mechanisms, grounding and subjective experience are different questions. There is no need to settle them before verifying a citation, respecting a tool permission or checking a program's result. Uncertainty here should sharpen the test, not excuse an unsupported claim.
A worked beginner check
Before opening the answer, explain why a system can produce a valid JSON answer with a correct citation ID and still be wrong. Then distinguish a changed prompt from an optimizer update.
Answer: JSON validates structure; citation membership validates an address; neither proves that the claim follows from that source. A prompt changes the input to a fixed parameterized computation. An optimizer update changes parameters using a training objective. Saved conversational memory changes later inputs without necessarily changing model weights.
For attention weights .75/.25 and values [2,0]/[0,4], the output [1.5,1] is a weighted mixture. A second head can read different values; it is not automatically a “reasoning head.” If the causal mask lets a training token read its target in the future, a low loss can be leakage rather than learned prediction.
Where to go from here
- Use: practise identifying tokens/context, unknowns, sources and tool boundaries; compare a factual claim with its primary evidence.
- Learn: study self-attention, then the complete block; calculate shapes rather than memorizing names.
- Build: use the 80-session map, beginning with arrays and gradients; later construct a tiny GPT and evaluate held-out data.
- Judge claims: contrast the primary critiques and exploratory capability reports, name their evidence type and propose a discriminating test.
The optional watch-list below retains the original breadth, but videos are supplementary learning aids, not the primary evidence for this corrected history. Video availability may change, and the suggested schedule is not a fixed mastery time. Read the associated written mechanism and solve its checkpoint even if you skip the video.
Watch-list — pick by how much time you have
Optional resources retained from the original; viewing and current metadata verification are outside this review.
If you have 20 minutes
-
Large Language Models explained briefly — 3Blue1Brown
-
What are Large Language Models (LLMs)? — Google for Developers
-
What ChatGPT Is Actually Doing Inside — CGP Grey
If you want the story
-
The 35 Year History of ChatGPT — Art of the Problem
-
The moment we stopped understanding AI [AlexNet] — Welch Labs
-
ChatGPT with Rob Miles — Computerphile
If you have two hours
-
[1hr Talk] Intro to Large Language Models — Andrej Karpathy
-
But what is a neural network? (Chapter 1) — 3Blue1Brown
-
AI Language Models & Transformers — Computerphile
If you have a weekend
-
Transformers, explained visually (Chapter 5) — 3Blue1Brown
-
Attention in transformers, visually explained (Chapter 6) — 3Blue1Brown
-
How might LLMs store facts? (Chapter 7) — 3Blue1Brown
-
Deep Dive into LLMs like ChatGPT — Andrej Karpathy
-
Transformer Neural Networks, clearly explained — StatQuest
-
Word2Vec, clearly explained — StatQuest
-
What does it mean for computers to understand language? — vcubingx
If you would rather build it
-
Let's build GPT: from scratch, in code, spelled out — Andrej Karpathy
-
Let's reproduce GPT-2 (124M) — Andrej Karpathy
-
Attention is all you need — implemented from scratch — Umar Jamil
For the disagreement
-
Yann LeCun: Meta AI, Open Source, Limits of LLMs, AGI — Lex Fridman Podcast #416
-
Attention Is All You Need (paper explained) — Yannic Kilcher
The primary sources
Start with the 2017 Transformer paper for that architecture family; it does not describe every later language model. These links are a primary reading route, not a claim that every linked secondary resource was revalidated.
| Year | Work | Why it matters |
|---|---|---|
| 1948 | A Mathematical Theory of Communication | Shannon develops information theory and statistical language approximations. |
| 2003 | A Neural Probabilistic Language Model | Bengio et al. — learned representations share statistical strength. |
| 2013 | Efficient Estimation of Word Representations | word2vec — embeddings at scale. |
| 2014 | Sequence to Sequence Learning | Sequence in, sequence out. |
| 2014 | Neural MT by Jointly Learning to Align and Translate | Learned soft alignment lets the decoder read source annotations during translation. |
| 2017 | Attention Is All You Need | The Transformer. The one to read. |
| 2020 | Scaling Laws for Neural Language Models | Why scale became a strategy rather than a gamble. |
| 2020 | Language Models are Few-Shot Learners | GPT-3 — learning from prompt examples alone. |
| 2021 | On the Dangers of Stochastic Parrots | The central critique. Read it even if you disagree. |
| 2022 | Training LMs to Follow Instructions | InstructGPT — a documented SFT, reward-model and PPO recipe. |
| 2023 | Sparks of Artificial General Intelligence | An exploratory early-GPT-4 capability study and the authors’ AGI interpretation. |
Friendlier secondary reading: The Illustrated Transformer for the architecture, Hugging Face on RLHF for alignment, and Jurafsky and Martin's Speech and Language Processing as the free standard textbook.
Pause / Recall / Apply
Can you explain it without the page?
Close the example. Reconstruct the core idea, then change one assumption. Mark complete when you’re ready; you can always undo it.
Stored in this browser only. No account, no sync. Clearing browser data removes your record.