In the last article we saw the shape of RAG: fetch the relevant passages, then let the model answer from them. All the interesting bits, and pretty much everything that makes RAG work well or poorly, live inside that first word — fetch. How does a system actually find the “relevant” passages, out of thousands or millions of them, given only your question?
The old answer was keyword search — match the words. But keywords miss a huge amount, because language is far too flexible for exact word matches to be enough. The modern answer is far more elegant, and it turns out to be one of the loveliest ideas in AI: search by meaning. This article is about how that works, using an idea we already met in the AI Primer — embeddings.
Why keyword search isn’t enough
Let me convince you that plain keyword matching wouldn’t cut it. Suppose your company’s HR document says:
“Employees are entitled to twenty days of annual leave per calendar year.”
And an employee searches: “How much time off do I get?”
Not a single significant word in the question appears in the answer. “Time off” isn’t “annual leave”; “how much” isn’t “twenty days”; “get” isn’t “entitled to.” Keyword search would return nothing helpful. Yet to any human, these two sentences are obviously about the same thing.
That gap — between word matching and meaning matching — is the whole problem. We want a search that understands the question is about the amount of leave and the answer contains the amount of leave, whatever words either happens to use. That’s semantic search, and it starts with embeddings.
The idea: turn meaning into location
Recall from the Primer’s article on word embeddings: we can turn a word into a list of numbers — its coordinates — that place it in an imaginary space where words with similar meanings sit close together. “Cat” and “kitten” become neighbours; “cat” and “helicopter” end up far apart. Meaning becomes location.
The beautiful extension for RAG is that we can do the exact same thing for whole passages. A sentence — or a paragraph, or a page — can be turned into a single point in that space, positioned so that passages about similar things sit close together. “Employees are entitled to twenty days of annual leave” and “How much time off do I get?” — despite sharing no significant words — end up as near neighbours in this meaning-space, because they’re about the same thing. Figure 1 shows this happening.
Figure 1: When passages are turned into points in an embedding space, ones that mean similar things end up close together — even if they share no words in common.
The list of numbers that represents a passage’s location is called its embedding. And the map itself — the imaginary space in which every possible passage has a location — is called the embedding space. Nothing mystical, just a way of turning meaning into geometry so a machine can measure it.
Searching by “how close in meaning”
Once every passage has an embedding, semantic search becomes wonderfully simple. When a question comes in, you do three quick things:
- Turn the question into its own embedding — its location in the same meaning-space.
- Find the passages whose embeddings are closest to the question’s — its nearest neighbours in meaning.
- Return those passages as your search results.
That’s the whole thing. You’re not looking for word overlap — you’re looking for meaning overlap, measured as closeness in the embedding space. Figure 2 shows this in action.
Figure 2: Semantic search. The question becomes a point in the embedding space, and the system returns the passages whose points sit nearest to it — its closest matches in meaning.
The measurement used to define “closest” has a name — cosine similarity — but the intuition is exactly the everyday one: which passages point in the same direction as the question in this meaning-space. Same direction, similar meaning. Different direction, different topic. In pseudocode it looks like this:
# Build the searchable collection ahead of time:
passage_embeddings = [embed(p) for p in passages]
# When a question arrives:
q_emb = embed(question)
scores = [similarity(q_emb, pe) for pe in passage_embeddings]
top_matches = pick_top_k(passages, scores, k=5)You compute a similarity score between the question and every passage, and hand back the top handful. That handful — those top-k matches — is exactly what gets fed into the language model for the “generate” half of RAG.
Where do embeddings come from?
You might be wondering: who decides these coordinates? Nobody, in the usual sense. Embeddings are produced by an embedding model — a small neural network specifically trained to place similar-meaning texts near each other in space. It learned this the same way we saw in the AI Primer, by being trained on huge amounts of text until it captured the patterns of what tends to go with what. In practice, you feed a piece of text into the embedding model, and it hands back the list of numbers.
Real-world embeddings are often several hundred numbers long (300, 768, 1536 are common). You can’t picture 1,536 dimensions and neither can I, but the maths of “close” and “far” works exactly the same as it does in the two-dimensional pictures. More dimensions just give the space more ways to distinguish subtle differences in meaning.
Why this is the engine of RAG
Look back at what we’ve done. We started with a question that shared no words with the answer, and we ended with a system that finds the answer effortlessly — because it’s searching by meaning rather than by words. The embedding is the bridge. It’s how a machine, which fundamentally operates on numbers, gets a real handle on what text is about.
This is why RAG works so well. It gives the language model a much better shot at finding the right material to answer with, because it can retrieve based on the intent of your question rather than the exact words you happened to use. And the same idea powers a surprising amount of what looks intelligent in modern AI tools — search, recommendations, clustering, deduplication, all rely on this same “meaning as location” trick.
But knowing how to find nearby embeddings is only half the practical story. If your document collection has millions of passages, checking the question against every one of them would be painfully slow. Modern RAG systems store embeddings in a specialised kind of database that can find nearest neighbours astonishingly fast — a vector database — and choosing and using one well is a genuine architectural decision. That’s next.
Next in the series: Vector Databases: How They Work & How to Choose — the specialised storage that makes embedding search fast at scale.