llmbeginner 18m read

Where LLMs Came From, and What They Actually Are

The 80-year origin story, the five ideas that actually matter, and a verified watch-list to go deeper — no maths required.

Most explanations of large language models start in the wrong place. They open with attention heads and softmax, and by paragraph three you are looking at a matrix. That is the inside of the machine. It tells you nothing about why the machine exists.

This post goes the other way. First the story — an 80-year chain where each step exists because the previous step hit a wall. Then the five ideas you genuinely need. Then a watch-list, so you can pick up the depth from people who explain it better than any article can.

No maths. If you can read a sentence and guess the next word, you already have the core intuition.

The one-paragraph version

A large language model is a program that predicts the next chunk of text. That is the whole job. You give it "The capital of France is" and it produces "Paris" — not because it looked anything up, but because across an enormous amount of text, that is overwhelmingly what came next. Everything else you have heard about — chat, reasoning, code, refusals — is a consequence of doing that one thing extremely well, at enormous scale, and then shaping the result.

Hold on to that. Every section below is either how we got there or what falls out of it.

Part 1 — How we got here

The useful way to read this timeline is as a chain of dead ends. Each idea solved the previous one's fatal flaw.

1948
Shannon asks a strange question
In 'A Mathematical Theory of Communication', Claude Shannon models English as a probability game — given the letters so far, what comes next? He generates gibberish that looks eerily like English. The entire field starts here.
1966
ELIZA fools people with no understanding
Joseph Weizenbaum's therapist chatbot works by pattern-matching and reflecting your words back. People confide in it anyway. First hard lesson: fluency and understanding are different things.
1980s–90s
n-grams, and the wall
Count which words follow which. Genuinely useful for speech recognition. But it breaks twice: context beyond a few words is unaffordable, and unseen phrases get probability zero. Words are just IDs — 'cat' and 'kitten' are as unrelated as 'cat' and 'bureaucracy'.
2003
Bengio: let the model learn what words mean
A neural language model represents each word as a learned vector instead of an ID. Similar words land near each other, so the model generalises to phrases it never saw. The wall cracks.
2013
word2vec makes it obvious
Mikolov's team trains word vectors cheaply at scale, and the geometry is startling — related words cluster, and simple arithmetic on vectors lands near sensible answers. Meaning as direction in space stops being theory.
2014
seq2seq — output a whole sequence
Sutskever, Vinyals and Le read a sentence into a single vector and generate another sentence out. Translation without hand-written rules. The flaw is brutal: the entire input must survive inside one fixed-size vector.
2014–15
Attention fixes the bottleneck
Bahdanau and colleagues let the decoder look back at every input word and weight what matters right now. Nothing has to be crammed into one vector any more. This is the hinge of the whole story.
2017
'Attention Is All You Need'
Vaswani et al. throw away the sequential recurrence and keep only attention. Suddenly the whole sequence is processed in parallel — which means GPUs can be used properly, which means scale becomes possible. The Transformer.
2018–2020
Scale turns out to be the feature
GPT, GPT-2, GPT-3. Same core recipe, more data and more parameters. GPT-3 starts doing tasks it was never explicitly trained for, just from examples in the prompt. Scaling laws make this predictable rather than lucky.
2022
RLHF makes it usable
Raw models predict text; they don't follow instructions. InstructGPT adds human feedback to teach helpfulness. A smaller tuned model is preferred over a 100x larger raw one. Then ChatGPT ships and the interface reaches everyone.
2023 →
Everything at once
Bigger context windows, images and audio, tool use, open-weight models you can run yourself, and a serious argument about what any of it means.

Part 2 — The five ideas that actually matter

1. Tokens — text becomes numbers

Models don't see letters or words. Text is chopped into tokens — common words are one token, rarer ones split into pieces. "unbelievable" might become "un", "believ", "able".

Roughly, one token is about ¾ of an English word. This is not trivia — it explains real behaviour. It is why models historically fumbled counting letters in a word (they cannot see the letters), why they are worse in languages that tokenise inefficiently, and why pricing is per token.

2. Embeddings — meaning as position

Each token becomes a long list of numbers: a vector. These are learned, not assigned, and they arrange themselves so related things end up near each other.

The important consequence: the model can handle a sentence it has never seen, because it has seen things nearby. That is the generalisation n-grams could never manage.

3. Attention — deciding what matters

Take the sentence: "The trophy didn't fit in the suitcase because it was too big."

What is "it"? You resolved that instantly using "trophy", "fit" and "big" — and ignored most of the sentence. Attention is that, mechanised. For every token, the model scores every other token for relevance right now, and blends them accordingly.

"Multi-head" means it does this several times in parallel with different notions of relevance — one head tracking grammar, another tracking subject matter. Stack this many times and you get a Transformer.

4. Next-token prediction — the only objective

Training is repetitive and dumb: hide the next token, guess it, measure the error, nudge the parameters, repeat billions of times.

The non-obvious part is what that forces. To predict well across the whole internet, a model has to pick up grammar, facts, translation, code syntax, argument structure, and formatting — not because anyone asked, but because each one reduces prediction error. Capability arrives as a side effect of compression.

5. Three stages, three different purposes

Pre-training

Where knowledge comes from

Predict the next token across a huge text corpus
Slow, enormously expensive, done rarely
Produces fluency and raw knowledge
Result continues text rather than answering you

Fine-tuning

Where the format comes from

Further training on curated question-and-answer examples
Teaches it to respond rather than ramble
Much cheaper than pre-training

Alignment (RLHF)

Where the manners come from

Humans rank competing answers
A reward model learns those preferences
The model is tuned toward them
Tone, refusals and helpfulness are shaped here

Most complaints about "the AI's personality" are complaints about stage three, not stage one.

The mental model in one sentence

An LLM is a very large pattern-completion engine that learned an internal model of how text works by being forced to guess the next token, and was then trained to aim that ability at answering you.

The five ideas, compressed

Tokens: text is split into chunks, not words — this explains a lot of odd behaviour.
Embeddings: meaning is represented as position in space, so similar things generalise.
Attention: for each token, the model decides which other tokens are relevant right now.
Next-token prediction: the only training objective — everything else is a side effect.
Three stages: pre-training gives knowledge, fine-tuning gives format, alignment gives manners.

What LLMs are not

These misconceptions cause the most real-world mistakes.

The assumption	What is actually happening
It looks up facts in a database	There is no database. Facts are diffused across billions of weights. This is exactly why it can be fluently, confidently wrong.
It learns from your conversations	Weights are frozen after training. It remembers within a conversation because the transcript is re-fed as input. Close the tab and it is gone.
It "understands" the way you do	Genuinely contested — see below. What is certain: it has no body, no persistence, no goals between messages.
It thinks, then writes	It produces one token at a time. "Reasoning" that helps is reasoning written into the output — which is why "think step by step" works at all.
Bigger is always better	Scaling helps predictably, but data quality, tuning and tooling often matter more. A well-tuned small model beats a sloppy huge one.

The argument worth knowing about

You will meet people who are certain in both directions. Both have real arguments.

"It is sophisticated pattern-matching." Bender, Gebru and colleagues argued in On the Dangers of Stochastic Parrots (2021) that these systems stitch together training text without reference to meaning, and that fluency invites us to imagine understanding that is not there. Yann LeCun has argued separately that text-only prediction is missing something fundamental — no grounding in the physical world, no persistent goals.

"Something more is happening." Microsoft researchers, in Sparks of Artificial General Intelligence (2023), documented GPT-4 solving problems that look poorly explained by memorisation. Their claim is not that it thinks like a person, but that "just autocomplete" undersells what emerges at scale.

The uncomfortable truth: we do not have an agreed definition of "understanding" precise enough to settle this, and interpretability research is still early. My own position is that "just predicting text" is accurate about the mechanism and misleading about the results — in the same way "just chemistry" is accurate about a brain.

Watch-list — pick by how much time you have

Every video below was checked to be live and correctly attributed on 2 August 2026.

The primary sources

If you only read one, make it the 2017 Transformer paper — everything current descends from it.

Year	Work	Why it matters
1948	A Mathematical Theory of Communication	Shannon frames language as prediction. The origin.
2003	A Neural Probabilistic Language Model	Bengio et al. — learned word vectors break the n-gram wall.
2013	Efficient Estimation of Word Representations	word2vec — embeddings at scale.
2014	Sequence to Sequence Learning	Sequence in, sequence out.
2014	Neural MT by Jointly Learning to Align and Translate	Attention is introduced.
2017	Attention Is All You Need	The Transformer. The one to read.
2020	Scaling Laws for Neural Language Models	Why scale became a strategy rather than a gamble.
2020	Language Models are Few-Shot Learners	GPT-3 — learning from prompt examples alone.
2021	On the Dangers of Stochastic Parrots	The central critique. Read it even if you disagree.
2022	Training LMs to Follow Instructions	InstructGPT — how RLHF made models usable.
2023	Sparks of Artificial General Intelligence	The strongest "something more is happening" case.

Friendlier secondary reading: The Illustrated Transformer for the architecture, Hugging Face on RLHF for alignment, and Jurafsky and Martin's Speech and Language Processing as the free standard textbook.

Where to go from here

1Use

Just use them well

Stop here and go practise. Knowing about tokens, context and confident-wrongness already puts you ahead of most users.

2Learn

Get the architecture

3Blue1Brown chapters 5 to 7, then the Illustrated Transformer, then the 2017 paper. In that order.

3Build

Build one yourself

Karpathy's 'Let's build GPT', typed out yourself rather than watched. It is a different experience.

4Argue

Judge the claims

Read Stochastic Parrots and Sparks back to back, then watch the LeCun interview. Disagreeing well requires both sides.