An interactive guide to large language models
ChatGPT, Claude and Gemini all rest on one surprisingly simple idea. Here it is, built up piece by piece, with small experiments you can poke along the way.
The best way to learn
Chapter 1 · Guess
Read this and notice what your brain does at the end: The cat sat on the…
You probably thought mat before you got there. You didn’t look it up. You have simply seen enough sentences like this one to know what tends to come next.
A language model plays exactly this game. Given some text, it gives every possible next piece of text a probability. mat is the favourite here, but floor and sofa are very much in the running. The model never produces an answer; it produces a distribution of answers.
Your turn. Pick any of the guesses on the stage. The model adds your pick to the end of the text, reads the whole thing again, and makes a fresh set of guesses for the next slot.
Every score on this page is written by hand. No real model runs here.
That is the whole trick behind ChatGPT, Claude and friends: guess, append, repeat. A paragraph is this loop run a few hundred times. The rest of this guide opens up the box in the middle: how does it come up with those guesses?
Chapter 2 · Tokens
Before the model can guess anything, the text has to become numbers. First it gets chopped into tokens: common chunks of text. Often a token is a whole word, sometimes a piece of one, sometimes a single punctuation mark.
Notice the spaces: they belong to the token that follows, so ␣cat and cat are different tokens. Each one is then swapped for its number in the model’s vocabulary, a fixed list that runs to around a hundred thousand tokens in recent models. From here on, the model only ever sees those numbers.
Try typing something. Common words stay whole. Rare words, names and typos get broken into fragments. This is also why models famously struggle to count the r’s in strawberry: they never saw the letters, only the chunks.
Chapter 3 · Vectors
A token ID is only a name tag: token 9,314 is no closer in meaning to 9,315 than to 12. So the model looks each ID up in a big table and swaps it for a vector, a long list of numbers. On the stage, each token’s vector is drawn as a column of cells: violet for positive numbers, blue-grey for negative ones.
Nobody writes those numbers. They start out random and get nudged during training until tokens used in similar ways end up with similar vectors.
Squash them down to two dimensions and the words sort themselves into neighbourhoods. Click a word to see who lives nearby.
Even directions can carry meaning. The step from man to woman points roughly the same way as the step from king to queen. This is a cartoon of a real, famous result; actual vectors have thousands of dimensions and are a lot messier.
Chapter 4 · Attention
So far each token has its own vector, looked up without any context. That is a problem. What does it mean in this sentence? On its own, anything. You only know by looking back at the other words, and that is exactly what attention lets the model do.
For every token, attention works out how much to borrow from each earlier token, then mixes that information into its vector. Here it leans hardest on animal, so its vector now carries something of the animal.
Now change the ending from tired to wide. Streets are wide; animals get tired. The same machinery now points it at street.
Click other words too. None of them can look ahead: while the model is writing, the future hasn’t been written yet.
A real model runs dozens of these in parallel, called heads, and each learns its own habit. Some are easy to describe, like tracking the previous word. Most are not, and researchers warn against reading any single pattern as the model’s “reasoning”.
Chapter 5 · Layers
Attention is one half of a transformer layer. The other half is a small neural network that works on each token separately, digesting what attention gathered. Big models stack dozens of these layers, one on top of the other.
Each layer doesn’t replace a token’s vector, it adds an adjustment to it. Watch the strip for the last word, the, shift a little on every floor. By the top it has soaked up the context: it now encodes something like “a cat is sitting on something”.
Only the final vector at the last position is used to guess what comes next. It gets compared against every token in the vocabulary, producing one raw score each. These scores are called logits.
Chapter 6 · Choose
Logits can be any number, positive or negative. A function called softmax turns them into probabilities that add up to 100%, keeping their order: the biggest score gets the biggest slice, and small gaps in score become large gaps in probability.
Now the model has to choose. Always taking the favourite makes text repetitive, so instead it rolls a weighted die. Temperature sets how heavily it is weighted: turn it down and the favourite nearly always wins; turn it up and long shots get a real chance.
Roll a few times, then try it hot and cold. Many apps also drop the long tail entirely, so truly odd words can never be drawn.
This is why asking the same question twice can get you two different answers. Nothing about the model changed between the two runs. Only the dice did.
Chapter 7 · Learn
We’ve treated the scores as given. They come from the model’s parameters: billions of numbers in the embedding table, the attention heads and every layer. Training is how they get set. Here’s a model early in training: it thinks floor is likelier than mat.
Training takes a real sentence, hides the next token and checks the guess. The worse the guess, the higher the loss. Then every parameter is nudged a tiny amount in whatever direction would have made mat more likely. Press train a few times and watch the loss fall.
Now repeat that for every token in an enormous pile of text. How enormous? On the stage, the small square is roughly all of English Wikipedia. The big one is the 15 trillion tokens Meta reported training Llama 3 on. A whole novel would be too small to draw.
Chapter 8 · Chat
A model trained only on raw text is a brilliant autocomplete, not an assistant. Ask it a question and it may just carry on the list of questions, because on the internet, questions often come in lists.
So there’s a second round of training: on example conversations where an assistant answers well, then on feedback about which of its own answers people preferred. The machinery doesn’t change. The habits do.
And the chat window? Behind the bubbles it’s still one long document. Your messages and its replies are stitched together with special marker tokens, and the model does what it has done all along: predict what comes next.
The cat sat on the
Fig. 1.1 An unfinished sentence.
Putting it together
Next time an AI answers you, you’ll know what’s underneath: a very large stack of arithmetic, tuned on a staggering amount of text, playing the guessing game one token at a time.
Whether that adds up to understanding is a genuinely open question, and “just predicting the next word” undersells what those layers have learned to do. But it isn’t a black box anymore. You’ve seen the parts.
A note on what you’ve seen
The scores, vectors, attention weights and tokenizer on this page are written by hand to make each idea visible. Softmax, sampling, top-p and the training step are the real formulas, applied to tiny numbers. No language model runs on this page. Real models add details we skipped, such as positional information, normalisation and batching.
The same hands-on approach, applied to your own data and tools.
Explore AI & data training