What Is NLP? Teaching Machines to Work with Language

We just saw that machines learn by finding patterns in examples. Numbers like test scores are easy because they already live in the machine’s world. But language is a different kind of problem, and teaching a machine to handle it is one of the hardest problems in computing.

The field that tackles it is called Natural Language Processing, or NLP. “Natural language” means the messy, human language you and I speak, as opposed to the tidy, rigid language of computer code. This article is about why that’s so hard, and how machines get a grip on it at all.

Why language is genuinely difficult

Let me convince you that this is a real problem, not a made-up one. Here’s a sentence:

“I saw her duck.”

Did the person watch someone lower their head to avoid something? Or did they see a duck that belonged to her? You can’t tell. Both readings are perfectly valid, and you’d only know which is meant from the surrounding conversation.

Your brain handles this so smoothly you barely notice the ambiguity. A machine has no such instinct. To it, “duck” is just a sequence of letters that could mean a bird or a movement, and it has to somehow learn which one fits from context.

Now pile on everything else language throws at us:

• The same word means different things. A “bank” by the river is not a “bank” that holds your money.
• Different words mean the same thing. “Big,” “large,” “huge,” and “enormous” are near-cousins.
• Meaning hides between the lines. “It’s a bit chilly in here” is often a polite request to close the window, not a weather report.
• Word order changes everything. “Dog bites man” and “man bites dog” use identical words and mean wildly different things.
• We break the rules constantly. Slang, typos, sarcasm, half-finished thoughts; humans understand each other anyway.

If you tried to teach a machine language by writing rules, the old way from the last article, you’d never finish. There are simply too many exceptions. This is the kind of messy problem where learning from examples is the only approach that stands a chance.

What NLP actually tries to do

NLP is a broad umbrella, but almost everything under it falls into two buckets: getting the machine to understand language coming in, and getting it to produce language going out. Figure 1 lays out the common tasks so you can see the shape of the field.

Figure 1: NLP tasks split broadly into understanding language (reading it in) and generating language (writing it out). Most real applications combine both.

On the understanding side, you’ll find tasks like:

• Sentiment analysis: is this review happy or angry?
• Classification: is this email spam, or which department should this complaint go to?
• Named entity recognition: pulling out the people, places, and dates from a paragraph.
• Search and question answering: finding the passage that actually answers what you asked.

On the generating side:

• Translation: English in, Japanese out.
• Summarisation: a long report in, a short paragraph out.
• Text generation: the writing that chatbots do.

A tool like a modern chatbot is really doing both at once: it understands your question and then generates an answer. Everything else in this series is, in a sense, building up the machinery to do these two things well.

How a machine gets a first grip on text

So how does a machine even begin? It can’t learn from raw letters the way it learned from test-score numbers. It needs a way to turn language into something it can measure and compare. That full journey is what the next few articles are about, but let me give you a taste with the simplest possible example.

Suppose I want to sort short messages as “positive” or “negative.” One naive starting point is to just count cheerful words versus grumpy words:

good_words = {"love", "great", "happy", "excellent", "wonderful"}
bad_words  = {"hate", "terrible", "awful", "bad", "worst"}

def simple_sentiment(text):
    words = text.lower().split()          # break the sentence into words
    score = 0
    for word in words:
        if word in good_words:
            score += 1                    # cheerful word: nudge up
        elif word in bad_words:
            score -= 1                    # grumpy word: nudge down
    if score > 0:
        return "positive"
    elif score < 0:
        return "negative"
    return "neutral"

print(simple_sentiment("I love this, it is wonderful"))  # positive
print(simple_sentiment("This is the worst, I hate it"))  # negative

This actually works for simple cases, and it shows the basic move: turn words into something countable. But notice how easily it breaks. Feed it “this is not good” and it happily says “positive,” because it sees the word “good” and completely misses the “not.” It has no sense of order, no sense of context, no real understanding.

That gap, between counting words and genuinely grasping meaning, is the story of modern NLP. Everything clever that’s come since, from word meaning to attention to transformers, exists to close it.

The leap that changed everything

For decades, NLP inched forward with hand-crafted rules and simple counting tricks like the one above. Progress was slow and the results were brittle.

What changed was the shift we keep coming back to: instead of hand-writing rules about language, we started letting machines learn language from large amounts of text, including books, articles, conversations, and web pages. Given enough examples, a machine can begin to pick up not just which words are cheerful, but which words tend to appear together, which ones play similar roles, and eventually behaviour that looks a lot like understanding.

But to learn from text, a machine first has to convert words into numbers, because numbers are the only thing it can actually do maths on. How we turn words into numbers is a deeper question than it first appears, and it’s where we’re headed next.

---

Next in the series: Representing Text as Numbers: the first real step in making language something a machine can compute with.