Word Embeddings & Vector Space

We left the last article with a wish. We wanted a way to represent words that captures their meaning, so that “cat” and “kitten” look related, while “cat” and “helicopter” look like strangers. We also wanted it to be compact, without the sea of zeros that one-hot encoding drowned in.

The idea that does this is called a word embedding. It quietly powers search engines, recommendation systems, chatbots, and the retrieval behind them. Let me build it up gently.

The core idea: meaning as location

Here’s the trick. Instead of giving each word a lonely one-hot switch, we give each word a list of numbers, its coordinates, that place it at a specific location in an imaginary space. And we arrange that space so that words with similar meanings sit close together.

That’s the whole idea: turn meaning into location. Words that mean similar things become neighbours; words that mean different things end up far apart.

Let me make it visual with just two coordinates. Imagine a simple map where one direction (left–right) roughly captures “small to large,” and another direction (bottom–top) captures “animal to vehicle.” Figure 1 shows a handful of words placed on such a map.

Figure 1: When words become coordinates, related words cluster together. “Cat” and “kitten” sit side by side; “car” and “truck” form their own neighbourhood far from the animals.

In Figure 1, “cat” and “kitten” huddle together in one corner, “dog” is nearby, and “car” and “truck” cluster off in a completely different region. The machine hasn’t been told these relationships. It’s placed the words so that closeness on the map means closeness in meaning.

Why more than two dimensions

My little map used two directions, but real embeddings use many more, often hundreds. That sounds intimidating, but the idea is exactly the same; there are just more ways for words to be similar or different.

Two dimensions can only capture two kinds of “similar” (say, size and animal-ness). But words relate along countless subtle axes: formal vs casual, positive vs negative, action vs object, and hundreds of shades we couldn’t even name. Giving each word a list of, say, 300 numbers lets the space capture all of that richness at once. We can’t picture 300 dimensions, nobody can, but the machine does the maths just fine, and everything we say about the 2D map still holds.

So an embedding is just a list of numbers, like this (shortened to four for readability):

embeddings = {
    "cat":    [0.8, 0.2, 0.9, 0.1],
    "kitten": [0.7, 0.3, 0.9, 0.1],   # very close to "cat"
    "dog":    [0.8, 0.3, 0.8, 0.2],   # also nearby
    "car":    [0.1, 0.9, 0.1, 0.8],   # far away, different meaning
}

Notice how “cat” and “kitten” have almost identical numbers, while “car” looks completely different. That closeness is meaning, encoded as location.

Measuring closeness

If meaning is location, then to ask “how similar are two words?” we just measure how close their points are. The most common measure is called cosine similarity, and don’t let the name scare you. The intuition is simple. It looks at whether two words point in the same direction from the centre of the space. Same direction means similar meaning (a score near 1); opposite directions mean unrelated or contrasting (a score near 0 or below).

Here’s how simple it is to compute:

import numpy as np

def similarity(a, b):
    a, b = np.array(a), np.array(b)
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

cat    = [0.8, 0.2, 0.9, 0.1]
kitten = [0.7, 0.3, 0.9, 0.1]
car    = [0.1, 0.9, 0.1, 0.8]

print(round(similarity(cat, kitten), 2))   # ~0.98  -> very similar
print(round(similarity(cat, car), 2))      # ~0.38  -> not similar

The machine now has a numerical sense of which words are related. That single ability, comparing meanings by comparing locations, is the engine behind “find me documents about this,” “recommend something like this,” and the retrieval systems that let chatbots search your own files.

If you want the one formula, cosine similarity is just:

similarity = (how much the two vectors overlap) / (their lengths multiplied)

which conveniently ignores how long the vectors are and focuses on direction, meaning rather than magnitude.

The example that reveals something deeper

Here’s the result that made researchers sit up, and it still delights me. Because words are now points in space, you can do arithmetic with meaning. The famous example:

king − man + woman ≈ queen

Take the coordinates for “king,” subtract the coordinates for “man,” add the coordinates for “woman,” and the point you land on sits right next to “queen.” Figure 2 shows this clearly.

Figure 2: The direction that leads from “man” to “woman” is the same direction that leads from “king” to “queen.” The space has quietly captured the idea of gender as a consistent movement, so subtracting “man” and adding “woman” turns “king” into “queen.”

What Figure 2 reveals is important: the embedding space has captured relationships, not just similarities. The “step” from man to woman is the same step as from king to queen, or uncle to aunt. Nobody programmed this. It emerged from letting a machine learn word locations by reading enormous amounts of text, because in real writing, “king” and “man” get used in parallel ways to “queen” and “woman,” and the machine picked that up.

Where these numbers come from, and where we’re going

You might be wondering: who decides the coordinates? The answer ties back to everything we’ve built. The machine learns them from data by reading mountains of text and noticing which words appear in similar contexts. Words that show up in similar company (“the ___ purred,” “the ___ chased the mouse”) end up with similar coordinates. It’s the “learn from examples” idea from Article 2, applied to word meaning.

Embeddings aren’t a separate gadget bolted onto AI. They’re the language that everything downstream speaks. When we get to attention and transformers later in this series, the thing flowing through those systems is embeddings: words as rich points in space, being compared, combined, and transformed.

But to understand how a machine learns those coordinates, and how it does anything sophisticated with them, we need to meet the workhorse behind modern AI: the neural network. That’s next, and despite its brainy name, its basic building block is simple.

---

Next in the series: Neural Networks: Neurons, Layers & Weights: the machinery that learns embeddings and much more.