Before a machine can turn words into meaningful numbers, it has to answer a question so basic it’s easy to skip right past: what counts as a unit of text in the first place? Is the unit a whole sentence? A word? A letter? The answer is “tokens,” and the process of chopping text into them is called tokenization. It’s the very first thing that happens to anything you type into a language model, and getting a feel for it explains a surprising number of quirks you may have noticed.
The obvious idea, and why it isn’t enough
The natural instinct is: a token is a word. Just split on the spaces. And honestly, that’s a fine place to start:
sentence = "I love learning about AI"
tokens = sentence.split()
print(tokens) # ['I', 'love', 'learning', 'about', 'AI']Five words, five tokens. Clean and simple. But splitting on spaces alone runs into trouble fast, and the trouble is worth understanding.
Think about a word like “unbelievable.” To you and me it’s obviously related to “believe,” “believable,” and “unbelievably.” But if we treat every whole word as one indivisible token, the machine sees “believe” and “unbelievable” as two totally unrelated blobs, missing the shared root sitting right there in plain sight.
Now think about the flip side. If every distinct word must be its own token, our list of known tokens has to include every word that exists: every name, every typo, every new slang term, “antidisestablishmentarianism” and all. That list would be impossibly huge, and the moment someone typed a word that wasn’t on it, the machine would be completely stuck, staring at an “unknown word” it had never catalogued.
So whole-word tokens leave us caught between two bad options: a vocabulary that’s impossibly large, or a machine that chokes on anything unfamiliar. Modern systems escape this trap with a clever compromise.
The clever compromise: pieces of words
Instead of whole words or single letters, today’s language models mostly break text into subword pieces: chunks that are often a whole common word, but sometimes just a fragment. Figure 1 shows what this looks like.
Figure 1: Common words stay in one piece, while rarer or longer words split into familiar fragments. “Tokenization” might become “token” + “ization”; “unbelievable” might become “un” + “believ” + “able.”
The rule of thumb these systems learn is practical: common words get their own token, and rarer words get built from smaller, reusable pieces. “The,” “cat,” and “run” are common enough to each be a single token. “Unbelievable” is rarer, so it gets assembled from “un,” “believ,” and “able”, pieces that also help spell hundreds of other words. Figure 2 shows this splitting up close.
Figure 2: A single long word is built from smaller, reusable fragments. The same “un” and “able” pieces that build “unbelievable” also help build “unusual,” “readable,” and countless others.
This solves both problems at once. The vocabulary stays a manageable size (typically a few tens of thousands of pieces), and the machine can handle any word it meets, even one it’s never seen, by spelling it out of familiar fragments. A brand-new slang word or an unusual name simply gets broken into pieces the model already knows.
Every token becomes a number
Once the text is split into tokens, each token is looked up in a fixed dictionary and swapped for its ID number. That’s the handoff to everything else: the model never truly sees text, only these token IDs.
# A tiny pretend dictionary mapping tokens to ID numbers
token_to_id = {"I": 40, "love": 1820, "learn": 616, "ing": 278, "AI": 9552}
tokens = ["I", "love", "learn", "ing", "AI"] # note: "learning" -> "learn" + "ing"
ids = [token_to_id[t] for t in tokens]
print(ids) # [40, 1820, 616, 278, 9552]Real tokenizers have a dictionary of tens of thousands of pieces rather than five, but the principle is exactly this: text in, a list of ID numbers out. Those numbers are what the model actually processes.
Why you should care about tokens
This might feel like plumbing, but tokens quietly explain a lot of things you may have wondered about.
“Tokens” are the unit of pricing and limits. When you hear that a model has a “context window of 128,000 tokens” or that an API “costs so much per thousand tokens,” now you know the unit being counted. A handy rule of thumb: in English, one token is roughly ¾ of a word, so 1,000 tokens is about 750 words. The model measures everything, including your input, its output, and the limit on how much it can consider at once, in these pieces.
It explains some odd behaviour. Ever notice a model struggle to count the letters in a word, or to spell something backwards? Part of the reason is that it doesn’t see the word as a string of letters at all. It sees a token, a single chunk. The letters inside are, in a sense, invisible to it. Tasks that seem easy to us can be awkward for a system that thinks in tokens, not characters.
Different languages tokenize differently. Languages the tokenizer saw a lot of during its creation tend to split into fewer, cleaner tokens; rarer languages get chopped into more pieces, which can make them more expensive and a little harder for the model to handle. It’s an invisible bias baked into this very first step.
The tiny step that shapes everything after
Tokenization is the doorway every piece of text walks through before a language model can do anything with it. It’s not glamorous, but it decides the fundamental units the model reasons in. As we’ve seen, it also influences cost, limits, and even which tasks the model finds easy or hard.
We now have text chopped into tokens and each token turned into an ID number. But an ID number, remember from the last article, is just an arbitrary label. It carries no meaning. The next article is about the moment meaning enters the picture, when those tokens are turned into rich numerical fingerprints that capture what words are about. That idea is embeddings.
Next in the series: Word Embeddings & Vector Space: turning tokens into numbers that capture meaning.