At the end of the last article we had a base model: a transformer that had played the guess-the-next-word game across a large body of text until it became a strong predictor, full of knowledge and fluency. And then I said something that might have surprised you: this powerful thing is not yet a helpful assistant. It’s autocomplete, not an assistant.
This article is about the steps that take a raw, knowledgeable-but-unruly predictor and shape it into the helpful assistant you actually talk to. It’s the difference between a model that knows things and a model that’s useful.
Why the base model isn’t enough
Let me make the problem concrete, because it’s easy to underestimate. Suppose you type a question into a raw base model:
“What’s a good way to stay focused while working?”
You’d expect helpful tips. But a base model might reply with something like:
“How do I stop getting distracted? Why is it so hard to concentrate? What apps help with focus?”
It answered your question with more questions. Why? Because it learned purely to continue text in a realistic way, and in its training data, a question is very often followed by more questions. Think of a list of FAQs, or a forum thread. The base model isn’t broken; it’s doing exactly what it was trained to do: predict plausible next text. It just was never taught that when you ask a question, you want an answer.
So the base model has two gaps. It doesn’t reliably know how to behave like a helpful assistant, and it hasn’t learned which behaviours we actually want: being helpful, honest, and safe rather than just plausible. Closing those two gaps takes two more stages of training. Together with pretraining, that makes three stages in all, shown in Figure 1.
Figure 1: Three stages. Pretraining builds a knowledgeable predictor. Supervised fine-tuning teaches it to act like a helpful assistant. Reinforcement learning from human feedback tunes it toward the responses people actually prefer.
Stage 2: Supervised fine-tuning: learning by good example
The first shaping step is called supervised fine-tuning, and the idea is exactly how you’d teach an apprentice: show them lots of good examples of the work done well.
We gather many examples of the desired behaviour: a question paired with a helpful answer, an instruction paired with a good response, and we continue training the model on these. It’s the same training loop as always (predict, check, adjust), but now the examples are curated demonstrations of being a good assistant, not just random internet text.
After enough of these, the model learns the pattern: when someone asks a question, provide a helpful answer; when someone gives an instruction, follow it. Our focus example now gets a proper reply with actual tips, not a volley of more questions. The model has learned the shape of being helpful.
This gets us most of the way. But “good behaviour” is subtle, and it’s hard to write out enough perfect examples to cover everything. There’s a limit to how far demonstrations alone can take us, which is where the third stage comes in.
Stage 3: Learning from human preferences (RLHF)
The final stage has a mouthful of a name: Reinforcement Learning from Human Feedback, or RLHF. Instead of showing the model the perfect answer every time (which is hard to produce), we let it try, and then tell it which of its attempts was better.
Here’s how it works. The model generates a few different responses to the same prompt. A person looks at them and simply ranks them: this one’s best, this one’s worse, this one’s unhelpful. Do this across many prompts, and you’ve gathered a big collection of human preferences: which kinds of answers people actually like.
Those preferences are used to train a second model, a reward model, whose job is to predict how much a human would like any given response. And then the main model is tuned to produce responses that score well with this reward model. In effect, the model learns to prefer responses humans rated highly. Figure 2 shows this loop.
Figure 2: The model produces several responses; a human ranks them; those rankings train a reward model that captures human preferences; the main model is then tuned to produce responses people prefer. The loop repeats.
The analogy I find most natural is coaching. A good coach doesn’t hand you a script for every situation. They watch you play and say “that choice was better than that one,” and over time you internalise their judgement. RLHF is similar: humans provide the judgement, and the model gradually absorbs a sense of what good looks like.
You don’t really need code for this one. It’s a process, not a formula. But the raw material is as simple as a pile of comparisons that look like this:
# The kind of preference data that trains the reward model:
preference = {
"prompt": "Explain gravity to a five-year-old.",
"better": "Gravity is what pulls things down, like when you drop a ball!",
"worse": "Gravity is a fundamental force described by general relativity...",
}
# A human judged the first response friendlier for the audience.
# Thousands of judgements like this teach the model what people prefer.What alignment actually buys us
This whole shaping process, supervised fine-tuning plus RLHF, goes by the broader name alignment: bringing the model’s behaviour into line with what humans actually want. It’s what makes a model:
- Helpful: it answers your question instead of dodging or rambling.
- Instruction-following: it does what you ask, in the format you ask for.
- Honest-leaning: it’s nudged toward accuracy and toward admitting uncertainty.
- Safer: it’s taught to decline genuinely harmful requests.
Alignment is the layer that separates the assistant you find useful from a raw predictor you’d find frustrating and strange. Most models you interact with have been through it.
A necessary note of humility
Here’s something important, and it sets up an article to come: alignment reduces bad behaviour, but it doesn’t eliminate it. The model is still, underneath all the polish, a next-word predictor optimised for plausible text. It can still confidently state things that are wrong, still be inconsistent, still be led astray. Alignment improves the behaviour a great deal, but it doesn’t give the model a reliable grasp of truth. We’ll dig into why in a couple of articles, when we look at why these models “hallucinate.”
For now, we have the full story of how a model comes to be: pretrained into a knowledgeable predictor, then fine-tuned and aligned into a helpful assistant. The natural next question is the one you experience every time you use one: when it writes back to you, how does it actually produce those words, one after another? That’s next.
Next in the series: How LLMs Generate Text: Sampling & Temperature: what actually happens each time the model writes you a reply.