In the last article we finished assembling the transformer: the full machine, every part in place. But I ended on a deliberately deflating note: a freshly built transformer knows nothing. Its millions of weights are random numbers, and if you fed it a sentence, it would babble nonsense. The architecture is ready, but the knowledge is not there yet.
This article is about how that empty architecture becomes something that can write essays, answer questions, and explain ideas: how a transformer turns into a large language model, an LLM. The process rests on one simple game, played at very large scale.
The game: guess the next word
Here’s the entire training task, and it really is this simple. We show the model a piece of text with the next word hidden, and ask it to guess what comes next.
“The cat sat on the ___”
The model predicts. Maybe it guesses “mat.” We check against the real text, see how it did, and, using the training loop from Article 9 (loss, gradient descent, backpropagation), nudge its weights to make a better guess next time. Then we do it again with a different snippet. And again. And again, billions of times, over a large collection of text: books, articles, websites, conversations, and other writing.
That’s it. That’s the whole “pretraining” process. The model plays guess-the-next-word over and over until it gets very good at it. Figure 1 shows the game.
Figure 1: The model reads the words so far, predicts the next one, checks against the real text, and adjusts. Repeated across a vast amount of writing, this single game is how a language model learns.
Why such a simple game teaches so much
When I first heard this, my reaction was: surely guessing the next word can’t teach it that much? But sit with it, and it turns out to teach a lot. Here’s the key insight: to predict the next word really well across broad writing, you’re forced to learn many patterns about language and the world.
Think about what it takes to fill in these blanks:
- “The capital of France is ___”: to guess “Paris,” you need a fact.
- “She was exhausted, so she went to ___”: to guess “sleep” or “bed,” you need everyday reasoning.
- “2 + 2 = ___”: to guess “4,” you need a bit of arithmetic.
- “The detective realised the murderer was ___”: to guess well, you need to have followed the logic of the story.
To get good at the next-word game across billions of examples, the model has to pick up grammar, facts, reasoning patterns, writing styles, and a rough working model of how things relate. Nobody hand-labels any of this. The text is its own answer key: the next word is always right there in the original, waiting to be checked against. This trick, where the data teaches itself with no human labelling required, is a big part of why this could be done at such scale.
Here’s a tiny, toy version you can actually run, to see the principle. It learns, purely from a handful of sentences, which word tends to follow another:
from collections import defaultdict, Counter
text = "the cat sat on the mat the cat ran to the mat"
words = text.split()
# Learn: after each word, which words tend to come next?
following = defaultdict(Counter)
for current, nxt in zip(words, words[1:]):
following[current][nxt] += 1
def predict_next(word):
return following[word].most_common(1)[0][0]
print(predict_next("the")) # 'cat' -> it learned "the" is often followed by "cat"
print(predict_next("cat")) # 'sat' -> and "cat" by "sat"This toy only remembers single-word pairs, so it’s very simple compared to a real model. But the principle is identical: learn, from large amounts of text, what tends to come next. A real LLM does this with the full power of the transformer, using attention to consider the entire context, not just the previous word, which is why it can predict the next word in much richer ways.
Where the “large” comes from: scale
If the game is so simple, why did powerful LLMs only appear recently? The answer is scale, and it comes in three flavours that all have to grow together. Figure 2 shows the three ingredients.
Figure 2: Capability emerges from scaling three things together: the amount of training data, the number of parameters (weights), and the amount of computing power used to train.
- Data: more text to learn from.
- Parameters: more weights, meaning a bigger, more capable model. This is where numbers like “7 billion” or “400 billion parameters” come from: they’re counting the tunable weights we met in Article 7.
- Compute: more raw computing power to run all those billions of guess-and-adjust cycles, which is why training big models is so expensive and hardware-hungry.
As you scale these three up together, the model doesn’t just get a bit better. New abilities can emerge. At smaller scales the model may barely string a sentence together; scale it up enough and it can summarise, translate, write code, and reason through problems, sometimes crossing thresholds where a skill that wasn’t there before starts to appear. The simple game, played at enough scale, produces much more capable behaviour.
What you get at the end: a base model
After all this pretraining, we have what’s called a base model: a transformer whose weights now capture an enormous amount of knowledge and linguistic skill. It is a spectacularly good next-word predictor.
But, and this matters, it is still just a next-word predictor. It’s a strong autocomplete, not yet a helpful assistant. If you typed a question into a raw base model, it might not answer at all; it might just continue with more questions, because in its training text, questions are often followed by more questions. It has knowledge and fluency, but it hasn’t learned to be helpful, to follow your instructions, or to behave itself.
Turning this raw, knowledgeable-but-unruly predictor into the helpful assistant you actually interact with takes one more shaping step. That step, fine-tuning and alignment, is where the base model learns how to respond to people, and it’s where we go next.
Next in the series: Fine-Tuning & Alignment: how a raw predictor becomes a helpful assistant.