The Transformer Architecture, End to End

This is the article the whole series has been building toward. We now have every ingredient we need: embeddings that capture meaning, self-attention that lets words look at each other, multiple heads for richness, activation functions and softmax, and the training loop that tunes it all. It’s time to assemble them into the transformer.

The transformer is the architecture behind most large language models in use today. Its name even hides inside the “T” of many model names. And the good news is that you already understand its hardest part: self-attention. What’s left is mostly a matter of arranging the pieces sensibly. Let me walk you through the whole machine, from the words going in to the prediction coming out.

First, a problem to fix: attention forgets order

There’s one gap we have to close before assembling anything. Self-attention has a blind spot: on its own, it doesn’t care about word order. It treats a sentence as a set of words all looking at each other, with no built-in sense of which came first. But we saw long ago that “dog bites man” and “man bites dog” mean opposite things. Order matters, and attention alone throws it away.

The fix is simple: before the words enter the machine, we add a little bit of information to each one that encodes its position in the sentence: first, second, third, and so on. This is called positional encoding. Now each word carries not just its meaning (from its embedding) but also its place in line, so attention can take order into account.

The transformer block: the repeating unit

The transformer is built from a single unit repeated over and over. Understand this one block, and you understand the whole thing, because the full model is just a stack of identical blocks. Each block does two things in sequence. Figure 1 shows it.

Figure 1: A single transformer block. Words first flow through multi-head self-attention (mixing in context from other words), then each word passes through a small feed-forward network (further processing). This block is stacked many times over.

Step 1: Multi-head self-attention. This is the mechanism from the last article. Every word looks at every other word and updates itself with the relevant context. After this step, each word is no longer an island. It “knows” about the words around it that give it meaning.

Step 2: A feed-forward network. After mixing in context, each word is passed through a small neural network of its own, exactly the neurons-and-layers-and-activation setup from earlier in the series. This step does some extra processing on each word individually, refining what attention produced.

Two small but important helpers wrap around these steps to make training work smoothly on very deep stacks. Residual connections let each step add its result to what came before rather than replacing it (so nothing important gets lost as information flows up), and layer normalisation keeps the numbers well-behaved so the whole tower trains stably. You don’t need the details here; just know they’re the support pieces that let us stack many blocks.

Here’s the whole block as simple pseudocode, so you can see how few moving parts there really are:

def transformer_block(words):
    # Step 1: let every word gather context from every other word
    attended = multi_head_self_attention(words)
    words = layer_norm(words + attended)          # add & normalise (residual)

    # Step 2: process each word a bit more, on its own
    processed = feed_forward(words)
    words = layer_norm(words + processed)          # add & normalise (residual)

    return words

That’s it. Attention, then feed-forward, each wrapped with an “add and normalise.” A whole transformer is just this block, stacked.

Stacking the blocks: understanding in layers

A single block enriches each word with context once. But stack many blocks on top of each other, with the output of one feeding into the next, and understanding builds up in stages. Early blocks tend to capture simple, local patterns; later blocks build on those to grasp more abstract, longer-range structure and meaning. It’s the same “layers build complexity” idea we saw with neural networks, now applied to language understanding. Big models stack dozens of these blocks.

The full journey, end to end

Let’s trace a piece of text through the entire machine, start to finish. Figure 2 shows the whole flow on one page.

Figure 2: The complete journey. Text is tokenized, each token becomes an embedding with positional information added, the result flows up through a stack of transformer blocks, and a final step produces probabilities for the next word.

Here is every step, and notice how it gathers up the whole series:

1. Tokenize. The input text is chopped into tokens (Article 5).
2. Embed. Each token becomes a meaning-rich vector, its embedding (Article 6).
3. Add position. Positional encoding stamps each token with its place in the sequence, so order isn’t lost.
4. Flow through the stack. The tokens pass up through many transformer blocks. In each, self-attention mixes in context and the feed-forward step refines it, layer after layer (Articles 11 and 12).
5. Produce scores. At the top, a final layer produces a raw score for every possible next token, tens of thousands of them.
6. Softmax to probabilities. Those scores become probabilities (Article 8): maybe “Paris” gets 70%, and the rest split the remainder.
7. Pick the next token. The model chooses from that distribution, and the whole cycle can repeat to generate the next word, and the next.

Read that list again and you’ll see it’s a tour of nearly everything we’ve built. The transformer isn’t a single clever trick. It’s an arrangement of the ideas from this series, assembled into a machine that reads text and predicts what comes next.

Why it won

It’s worth naming why this design became so widely used. It connects any two words directly, no matter how far apart, so nothing fades (attention). It processes all the words in parallel, so it trains fast on huge data (attention again). And it stacks cleanly into deep, powerful models that keep getting better as you make them bigger. That combination, parallel and scalable, is why the transformer became the foundation for today’s large language models.

From architecture to intelligence

We’ve now built the machine. But a freshly assembled transformer, with its weights still random, knows nothing. What transforms this architecture into something that can write, answer, and reason is training it on a large amount of text, until its weights capture patterns in human language. That process, how a transformer becomes a large language model, is where we head next.

---

Next in the series: From Transformer to LLM: Pretraining at Scale: how this architecture is trained into the AI systems you actually use.