There’s a story hiding behind every friendly, helpful chatbot you’ve ever used. The raw language model at its heart was, when it came out of pretraining, an unruly next-word predictor. It knew a lot. It wrote fluently. And it wouldn’t reliably answer your questions, follow your instructions, or behave itself in any of the ways you’d expect. Turning that raw model into the polite assistant you actually talk to takes two more shaping steps, and they have names worth knowing: instruction tuning and RLHF.
We touched on both back in the AI Primer, at a high level. In this article I want to open them up properly, because they’re the fine-tuning-family techniques that give models their character, and understanding them well changes how you think about every language model you use, whether you’re building one or just relying on one.
Where we left it in the Primer
Quick recap so we’re on the same page. A base model at the end of pretraining is a knowledgeable-but-unhelpful predictor. Ask it a question and it might well continue with more questions, because in its training text, questions are often followed by more questions. It hasn’t learned to be helpful, it’s just learned to produce plausible text.
To fix this, we saw that model builders do two more shaping steps. First, supervised fine-tuning on instruction-following examples (which is what “instruction tuning” refers to). Second, reinforcement learning from human feedback (RLHF), where the model is nudged toward responses people actually prefer. Together, these two are what turn the raw predictor into a real assistant, a process broadly called alignment.
Now let’s unpack each one and see how they actually work.
Instruction tuning: teaching the shape of helpfulness
Instruction tuning is a specific use of supervised fine-tuning, the same technique from the last few articles. What makes it “instruction tuning” is what’s in the dataset: thousands of examples of instructions paired with good responses to those instructions. Not one task, not one style, but a broad, varied collection covering all the different shapes of things people ask of an assistant.
A typical dataset might look like this:
[
{"instruction": "Summarise this article in 3 sentences.\n\n[article text]",
"response": "The article argues that..."},
{"instruction": "Translate the following to French: 'Where is the library?'",
"response": "Où est la bibliothèque ?"},
{"instruction": "Write a Python function that returns the largest of three numbers.",
"response": "def largest_of_three(a, b, c):\n return max(a, b, c)"},
{"instruction": "Give me three ideas for a birthday gift for a friend who loves hiking.",
"response": "1. A lightweight ..."}
]Notice what the dataset is teaching, aside from the specific answers. It’s teaching a pattern: when the user gives an instruction, respond with a helpful answer that follows the instruction. After seeing tens of thousands of pairs like this across every kind of task, the model absorbs the general shape of being helpful. That shift, from “predict plausible text” to “follow the user’s instructions,” is what instruction tuning delivers. Figure 1 shows the before and after.
Figure 1: Given the same question, a base model may continue with more questions (its training data taught it to). After instruction tuning on many examples of instruction-and-answer pairs, the same model reliably follows the instruction and produces an answer.
This is a milestone. After instruction tuning, the model is a usable assistant. It’ll follow your requests, answer your questions, and behave roughly the way you’d expect. Many open “instruct” or “chat” models you’ve heard of (say, Llama-3.1-8B-Instruct) are named that way because they’ve been through this step. They’re the base model, plus supervised fine-tuning on instructions.
You can see, if you followed the last few articles, that this is the fine-tuning technique we’ve already covered. Instruction tuning is supervised fine-tuning with a specific kind of dataset. The mechanics we walked through, LoRA and all, are often used to produce these instruction-tuned models today. So if you understood the earlier articles, you already understand the mechanics of instruction tuning.
But there’s a ceiling supervised alone can’t cross
Instruction tuning gets you far. It’s often enough on its own for a very useful assistant. But there’s a subtle limitation that becomes obvious once you notice it: it can only teach the model to imitate the specific examples in its dataset. And “being helpful” is subtler than any dataset can cover.
Consider two possible responses to the same question:
- Response A: technically correct, but curt, cold, and unhelpful in tone.
- Response B: technically correct, warm, patient, and phrased in a way that’s useful.
Both are “correct.” A supervised dataset can’t easily teach the model to prefer B over A, because both look like acceptable answers when judged one at a time. What you’d want is a way to tell the model, “when you have several plausible responses, pick the one people would actually prefer.” That’s the gap RLHF is designed to fill.
RLHF: learning from preferences, not scripts
RLHF stands for Reinforcement Learning from Human Feedback, which is a mouthful. The core idea is practical. Instead of showing the model the ideal answer, we let it generate several candidate answers, ask a human to rank them (from best to worst), and then use those rankings to nudge the model toward the kinds of responses humans prefer. Coaching, not scripting. Figure 2 shows the loop.
Figure 2: The RLHF loop. The model generates several candidate responses to a prompt. A human ranks them from best to worst. Those rankings train a small reward model that captures human preferences. The main model is then tuned to produce responses that score well according to the reward model. Repeat.
Reading it in a bit more detail:
- Generate several candidates. For a given prompt, the model produces two or three different responses (varying sampling or temperature gives you variety).
- Have a human rank them. A person compares the responses and picks which is best, worst, and in the middle. This is much easier for humans than writing ideal answers from scratch, which is one of the useful things about RLHF: humans are good at judging even when they’d struggle to produce.
- Train a reward model. All those human rankings are used to train a smaller “reward model,” whose job is to predict how much a human would prefer any given response. It’s essentially learning human taste.
- Tune the main model toward higher reward. The main model is then further trained to produce responses that score well according to the reward model. Not against fixed correct answers, but against learned human preferences.
The result is a model that has learned what humans prefer, in a way a dataset of scripted answers may not manage. Warmth, honesty, staying on task, admitting uncertainty, avoiding pointless hedging: all of these can be nudged in through the preference signal, when they’re too fuzzy to be captured by supervised examples alone.
Newer variants worth knowing about (briefly)
Because RLHF requires training a separate reward model and then running reinforcement learning on top of it, it’s operationally complex and expensive. So the field has been busy inventing simpler variants that keep the core idea, learning from preferences, but skip some of the machinery. The one you’ll hear about most is called DPO (Direct Preference Optimisation).
You don’t need the details, but the idea is worth naming: instead of training a separate reward model and then optimising against it, DPO takes the preference data directly and tunes the model against it in one step. Same fundamental idea (learn from ranked preferences), simpler implementation, often comparable results. Many recent open-model releases use DPO or its cousins instead of classical RLHF.
The takeaway is that the field is converging on preference-learning in some form, because it’s a strong way to teach the fuzzy qualities that separate “technically correct” from “good.”
Where all this leaves your fine-tuning project
So how does this connect to what you might do? Two useful framings.
Most people don’t need to do RLHF themselves. When you start with an instruction-tuned model (a -Instruct or -Chat variant), you’re already inheriting the full alignment work that model’s makers did. Your own fine-tuning, layered on top with LoRA, is almost always supervised fine-tuning with a narrow, focused dataset. That’s the right shape of the problem for most business use cases, and it’s what we walked through last article.
But when you do need it, you now know why. Some projects benefit from preference-based tuning: when you need consistent quality of judgement rather than consistent format, when the “right” answer is fuzzy, or when you want to reduce subtle failure modes that clear-cut examples can’t easily catch. If you find yourself in that territory, you’re looking at a DPO-style workflow: gather ranked preferences, tune against them, evaluate carefully.
From producing a model to knowing it’s good
Look at what we’ve built up in this track so far. You know when to fine-tune (article 1), what fine-tuning does (article 2), how to prepare data for it (article 3), the efficient techniques that make it practical (article 4), how to actually run it (article 5), and now the deeper alignment techniques that shape what kind of assistant you’re producing.
Every one of these steps ends with the same question: is the fine-tuned model actually better? We’ve touched on evaluation in passing, but it deserves its own full treatment, because a fine-tuning project that doesn’t evaluate rigorously is a fine-tuning project you can’t trust. That’s what the next article is entirely about.
Next in the series: Evaluating a Fine-Tuned Model, proving it’s actually better, not just newer.