Everything we’ve built so far handles inputs of a fixed shape: a photo with a set number of pixels, a row of numbers. Feed it in all at once, get an answer out. But language doesn’t work like that. A sentence is a sequence: the words arrive one after another, order matters enormously, and the meaning of each word depends on the ones around it. “Dog bites man” and “man bites dog” contain the same words in a different order and mean opposite things.
So to handle language, we need networks that respect sequence and context. This article is about the first clever attempt to do that, a design called the recurrent neural network, or RNN, and, just as importantly, about why it struggled. Understanding its limitations sets up the breakthrough that modern language models are built on.
The core idea: read one word at a time, and remember
Think about how you read a sentence. You don’t absorb all the words at once in a jumble. You read them one at a time, left to right, and as you go, you keep a running sense of what the sentence is about. By the time you reach the end, your understanding has been shaped by everything that came before.
An RNN copies this strategy. It processes a sentence one word at a time, and it carries a little bundle of information, call it its memory, from each word to the next. At every step it does two things: it looks at the current word and its memory-so-far, and from those it produces an updated memory to carry forward. Figure 1 shows this chain.
Figure 1: The RNN reads the sentence one word at a time. At each step it combines the current word with the memory carried over from the previous words, producing an updated memory that flows on to the next step.
That carried-along bundle (its proper name is the “hidden state”) is the RNN’s whole trick. It’s a form of memory that lets information from early words influence how later words are understood. In loose pseudocode, the process looks like this:
memory = start_empty() # begin with a blank memory
for word in sentence: # read one word at a time
memory = update(memory, word) # blend the new word into what we remember
answer = read_out(memory) # the final memory captures the whole sentenceNotice the word memory appears on both sides of that update line. The new memory is built from the old memory plus the new word. That looping-back is why it’s called recurrent. For a while, RNNs were the state of the art for language, translation, and speech.
Where it falls apart: the fading memory
Here’s the problem. That single bundle of memory has to carry everything important from the entire sentence so far, squeezed into a fixed amount of space. And with each new word, the memory gets updated, which means older information gradually gets written over and fades.
For a short sentence, fine. But consider something longer:
“The keys that I left on the kitchen table this morning, next to the pile of unopened mail and yesterday’s newspaper, are still there.”
To choose “are” (not “is”), you need to remember, all the way back at the start, that the subject was “keys” (plural). By the time an RNN has read through all the words in between, the memory of “keys” has been updated so many times that it’s faded to a whisper. The connection between “keys” and “are”, words that belong together but sit far apart, is exactly what the RNN tends to lose. Figure 2 shows this fading.
Figure 2: As the RNN reads on, information from the earliest words fades with every update. By the end of a long sentence, the beginning has nearly disappeared, so long-range connections get lost.
This is the famous weakness of RNNs, sometimes called the “long-range dependency” problem. They’re fine at connecting nearby words but struggle to link words that are far apart, and language is full of far-apart connections. (Engineers invented cleverer RNN variants with names like LSTM that helped the memory hold on longer, and they pushed things forward, but they never fully solved it.)
There’s a second, more practical problem too. Because an RNN reads strictly one word at a time, each step waiting for the one before it, it can’t easily be sped up by doing things in parallel. Feed it a long document and it plods through sequentially. In an era where we wanted to train on enormous amounts of text using hardware that thrives on doing many things at once, this sequential bottleneck was a serious handicap.
Two problems, one wish-list
So the RNN gave us something useful, a way to handle sequences with a memory of context, but it came with two stubborn flaws:
- Fading memory: it loses connections between words that are far apart, and language depends on those connections.
- No parallelism: it must read one word at a time, which makes it slow to train at scale.
Put those together and you can write the wish-list for whatever comes next. We want something that can (a) connect any two words directly, no matter how far apart they sit, without relying on a fading memory to ferry the information across, and (b) look at all the words at once rather than plodding through them one by one.
That wish-list turned out to describe attention, a mechanism that lets every word look directly at every other word and decide, for itself, which ones matter. That’s where we go next.
Next in the series: The Attention Mechanism: the idea that let models connect any two words directly.