Header image

Every time you send a message to an AI assistant and watch a fluent paragraph appear, something specific is happening behind the scenes. The model isn’t writing the way you and I do, planning a sentence and then setting it down. It’s producing text one small piece at a time, guessing each next token, then looking at everything so far and guessing again.

This article is about that generation process: how a model turns its next-word predictions into whole responses, and about a single dial, temperature, that controls whether its writing comes out focused and predictable or loose and creative. Understanding this gives you practical control over these tools.

Writing one token at a time

Recall from the transformer article that the model’s fundamental output is a prediction of the next token, expressed as probabilities (thanks to softmax). Generating a full response is just that single ability, run in a loop.

Here’s the loop. The model looks at everything so far, predicts the next token, adds it to the text, and then feeds the whole thing, now one token longer, back in to predict the token after that. It keeps going until it produces a special “I’m done” signal or hits a length limit. Figure 1 shows this cycle.

Generating text one token at a time Figure 1: The model predicts the next token, appends it, then feeds the longer text back in to predict the next one. This loop repeats, building the response token by token, left to right.

So if you ask it to complete “The cat sat on the,” it might generate:

  • Looks at “The cat sat on the” → predicts “mat” → text is now “The cat sat on the mat”
  • Looks at “The cat sat on the mat” → predicts “.” → and decides it’s done.

Responses from these models are built this way: no whole-sentence planning, just one token, then the next, then the next. (This is also why you often see the text stream out word by word: you’re watching the loop run.)

How does it choose each token?

At each step, the model doesn’t have just one candidate for the next token. It has a probability for every possible token. After “The capital of France is,” it might assign “Paris” 85%, “a” 5%, “home” 2%, and tiny slivers to thousands of others. So how does it pick one?

There are two broad strategies, and the difference between them matters.

Always pick the top choice (greedy). The simplest approach: at every step, choose the single most probable token. This is consistent and safe, but it can make the writing feel flat, repetitive, even robotic. Always taking the most obvious path never surprises anyone.

Pick randomly, weighted by probability (sampling). Instead of always grabbing the top token, roll a weighted die: a token with 85% probability gets chosen about 85% of the time, but sometimes a less likely token gets its turn. This makes the writing more varied and natural. Most of the text you see is generated with some form of sampling.

Temperature: the creativity dial

This is where temperature comes in: a single number that controls how adventurous that weighted die is. It’s probably the most useful knob to understand as a user.

  • Low temperature (near 0): the model sharpens its focus onto the top choices. It becomes predictable, consistent, and conservative. Perfect for facts, code, or anything where you want the safe, likely answer.
  • High temperature (around 1 or above): the model flattens the odds, giving unlikely tokens a real chance. It becomes creative, varied, and surprising, but also more prone to wandering off or saying something odd. Good for brainstorming, poetry, or fiction.

Figure 2 shows what temperature does to the probabilities.

How temperature reshapes the probabilities Figure 2: Low temperature sharpens the distribution toward the top choice (focused, predictable). High temperature flattens it, giving less likely tokens more chance (varied, creative).

Mechanically, it’s simple: before applying softmax, the model divides all the scores by the temperature. Dividing by a small number (low temperature) exaggerates the gaps, so the top choice dominates. Dividing by a larger number (high temperature) shrinks the gaps, so the field levels out. Here it is in code:

import numpy as np

def softmax_with_temperature(scores, temperature):
    scores = np.array(scores) / temperature   # the whole trick: divide by temperature
    e = np.exp(scores - np.max(scores))
    return e / e.sum()

scores = [3.0, 1.0, 0.5]   # raw scores for three candidate next tokens

print(np.round(softmax_with_temperature(scores, 0.3), 2))  # [0.99 0.   0.01] focused
print(np.round(softmax_with_temperature(scores, 1.5), 2))  # [0.62 0.21 0.17] varied

Look at the difference. At a low temperature of 0.3, the top token gets 99% of the probability, so the model will almost always pick it. At a higher temperature of 1.5, that drops to 62%, and the other options get a real look-in. Same underlying scores; completely different writing behaviour. In one line of maths, it’s just softmax(scores ÷ temperature).

(There are a couple of related dials you might bump into, names like top-k and top-p, that also shape the pool of candidate tokens the die rolls over. They’re refinements on the same idea: controlling how much variety the model is allowed. Temperature is the one worth knowing first.)

Why this is genuinely useful to know

This isn’t just trivia. It hands you a practical lever. When you want reliable, factual, repeatable output, such as extracting data, writing code, or answering a precise question, you (or the app you’re using) want a low temperature, so the model plays it safe. When you want fresh ideas, varied phrasings, or creative writing, a higher temperature lets it loosen up. Many tools expose this setting directly, and knowing what it does turns a mysterious slider into a deliberate choice.

It also explains a behaviour you’ve probably noticed: ask the same model the same creative question twice and you often get two different answers. That’s sampling at work, with the weighted die landing differently each time. At a temperature of zero, you’d get far more consistent (and far more repetitive) results.

The generation loop, in a sentence

So here’s the whole thing distilled: a language model writes by predicting one token at a time, sampling each from its probabilities, with temperature controlling how boldly it strays from the safest choice. That loop, running over and over, produces everything from a one-line answer to a full essay.

There’s one practical constraint we’ve been ignoring, though. This loop feeds “everything so far” back into the model at each step, but there’s a limit to how much text the model can actually hold in view at once. That limit shapes long conversations, costs, and a lot of how these tools behave in practice. It’s called the context window, and it’s next.

Next in the series: Context Windows & Tokens in Practice: the model’s working memory, and why its size matters.