We ended the last article with a wish-list. We wanted a way for a model to connect any two words directly, no matter how far apart they sit, without relying on a memory that fades as it travels. The idea that does this is called attention. Every modern language model, including the ones behind today’s chatbots, is built on it.
The core idea is something you do all the time, without noticing. Let me show you.
The idea you already use every day
Read this sentence:
“The trophy didn’t fit in the suitcase because it was too big.”
What does “it” refer to: the trophy, or the suitcase? You know instantly: the trophy. But pause and notice how you knew. To make sense of “it,” your mind quietly reached back through the sentence, glanced at the other words, and focused on the most relevant one, “trophy”, while paying less attention to the rest.
That act of focusing on the relevant words is attention. Change one word:
“The trophy didn’t fit in the suitcase because it was too small.”
and suddenly “it” means the suitcase. Same sentence structure, but your focus shifts to a different word based on meaning. You did that automatically. Attention is the mechanism that gives a machine this same ability: for any word, look at all the other words and decide which ones to focus on.
How it works: scores, weights, and a blend
Let me turn that intuition into something a machine can compute. It happens in three steps, and none of them is complicated.
Step 1: Score every other word for relevance. For the word we’re currently trying to understand (say “it”), the model compares it against every other word in the sentence and gives each a relevance score. “Trophy” scores high; “suitcase” scores lower; “the” scores near nothing.
Step 2: Turn those scores into weights. Raw scores are awkward, so we pass them through softmax, the exact function from Article 8, which turns them into clean weights that are all positive and add up to 1. Now we have something like: trophy 0.7, suitcase 0.2, everything else sharing the last 0.1. These weights say how much to focus on each word.
Step 3: Blend the words according to the weights. Finally, the model builds an updated understanding of “it” by mixing together the other words in proportion to their weights: mostly “trophy,” a bit of “suitcase,” a sprinkle of the rest. The result is a version of “it” that’s now infused with the meaning of “trophy.”
Figure 1 shows this in action.
Figure 1: To interpret “it,” the model draws a weighted connection to every other word. The thick, bright line to “trophy” (weight 0.7) shows where the focus lands; thinner lines to other words carry less weight. The weights add up to 1.
Here’s the whole thing in simple code. Imagine we’ve already scored how relevant each word is to “it”; attention just turns those scores into weights and blends the words:
import numpy as np
def softmax(x):
e = np.exp(x - np.max(x))
return e / e.sum()
words = ["trophy", "suitcase", "because", "big"]
scores = np.array([3.0, 1.5, 0.2, 0.5]) # how relevant each word is to "it"
weights = softmax(scores)
print(dict(zip(words, np.round(weights, 2))))
# {'trophy': 0.74, 'suitcase': 0.17, 'because': 0.05, 'big': 0.05}
# -> the model focuses ~74% on "trophy" when interpreting "it"If you want the idea in one line of maths, attention is: output = sum(weight_of_each_word × that_word), where the weights come from softmax over the relevance scores. A weighted blend, where the weights are learned focus. That’s the entire mechanism.
Why this beats the fading memory
Look at what attention fixed. An RNN had to ferry the meaning of “trophy” step by step, word by word, all the way to “it”, and we saw how that memory fades over distance. Attention avoids that problem. It lets “it” reach out and look at “trophy” directly, in a single move, no matter how many words sit between them. Near or far, it’s the same one step.
And there’s a second win. Because attention scores every word against every other word as a set of independent comparisons, all of them can be computed at the same time rather than one after another. That answers the second half of our wish-list: no more plodding sequentially through the sentence. Attention is naturally parallel, which is what let these models be trained on enormous amounts of text using hardware built to do many things at once.
The relationships are learned, not hand-coded
One more thing worth pausing on, because it’s easy to miss. Nobody programs the rule “when you see ‘it’, look at ‘trophy’.” The model learns, through the training process from Article 9, how to produce good relevance scores, and how to tell which words should attend to which. After seeing enormous amounts of text, it discovers on its own that pronouns tend to refer back to certain nouns, that adjectives attach to the things they describe, that verbs connect to their subjects. Attention is the mechanism; the training is what teaches it where to point.
Why it matters
Attention gave models the two things RNNs couldn’t: direct connections between any words, and the ability to process everything in parallel. That’s why nearly every powerful language model built since rests on it.
But so far I’ve described attention a little loosely: one word “looking at” the others. In the real design, this idea is made precise and applied within a sentence so that every word attends to every other word at once, each building a context-aware version of itself. That refined version is called self-attention, and paired with a trick for looking at several kinds of relationships at the same time, it’s the core mechanism inside the transformer. That’s next.
Next in the series: Self-Attention & Multi-Head Attention: the precise form of attention that powers the transformer.