Activation Functions & Softmax

In the last article we built a neuron: it takes inputs, multiplies each by a weight, adds them up with a bias, and passes along the result. Then we stacked neurons into layers. It seems like we’re basically done. Surely we can just keep stacking these little calculators and learn anything?

There’s a catch. As things stand, stacking these neurons doesn’t actually buy us the power we think it does. Fixing that requires one small extra ingredient, and adding it is what turns a network from a stack of straight lines into something that can learn complex patterns. That ingredient is the activation function, and its close cousin softmax is how a network turns its final answer into a clean choice. Let me explain both.

The problem: too many straight lines make a straight line

Here’s the tricky bit. A neuron computes a weighted sum, and a weighted sum is a straight-line relationship. If you feed the output of one weighted sum into another weighted sum, and that into another, you’d hope to build up something rich and complex.

But mathematically, a straight line followed by a straight line followed by a straight line is… still just a straight line. Stack a hundred of them and you still only get a straight line. All that depth, and the network can only draw straight relationships, which means it could never learn the curved patterns that real problems, like language or images, are full of.

We need to introduce a bend. Something that breaks the “straight line in, straight line out” chain and lets the network curve. That’s what an activation function does.

The fix: a little bend after each neuron

An activation function is a simple rule applied to each neuron’s output before it’s passed on. It takes the weighted sum and reshapes it slightly, adding the crucial bend. Figure 1 shows where it sits: right after the weighted sum, as the last step inside the neuron.

Figure 1: The activation function is the final step inside a neuron. After the weighted sum, it applies a simple “bend”; here, one that flattens all negative values to zero before passing the result on.

The most popular activation function today is blunt. It’s called ReLU, and its entire rule is: if the number is negative, make it zero; otherwise, leave it alone.

def relu(x):
    return max(0, x)

print(relu(3.5))   # 3.5  -> positive, passes through
print(relu(-2.0))  # 0    -> negative, flattened to zero

In one line of maths: ReLU(x) = max(0, x). That’s the whole thing. It looks almost too simple to matter, but that little kink, flat below zero and straight above it, is enough of a bend that, once you stack many of them, the network can approximate many kinds of patterns. You can think of each neuron as a tiny switch: below a certain point it stays quiet (zero), and above it, it speaks up. Layers of these switches, turning on and off in different combinations, are what give a network its expressive power.

There are other activation functions with different shapes. One called sigmoid gently squashes any number into the range between 0 and 1, which is handy when you want an output that reads like a probability. But the key idea is the same for all of them: add a bend, so the network can learn curves instead of only straight lines.

Softmax: turning raw scores into a clean choice

Activation functions live inside the network. Softmax usually lives at the very end, and it solves a different, very practical problem: how to turn a network’s raw output into a confident choice among options.

Imagine a network that looks at a photo and has to decide: cat, dog, or bird? Its final layer spits out three raw scores, say [2.0, 1.0, 0.1]. Higher means “more likely,” so cat is winning. But these raw numbers are awkward. They don’t add up to anything meaningful, some might be negative, and “2.0” doesn’t tell us how sure the network is.

What we’d really like is to convert those scores into probabilities: three positive numbers that add up to 1 (that is, to 100%), so we can say “80% cat, 15% dog, 5% bird.” That’s exactly what softmax does. Figure 2 shows the transformation.

Figure 2: Softmax takes a list of raw scores and turns them into probabilities that are all positive and add up to 1 (100%). The biggest score gets the biggest slice, and the gap is exaggerated so the winner stands out.

Here it is in code:

import numpy as np

def softmax(scores):
    exp_scores = np.exp(scores)          # make everything positive, exaggerate gaps
    return exp_scores / np.sum(exp_scores)  # rescale so they add up to 1

scores = [2.0, 1.0, 0.1]                  # raw scores for [cat, dog, bird]
probs = softmax(scores)
print(np.round(probs, 2))                 # [0.66 0.24 0.1 ] -> adds up to 1.0

The recipe has two steps. First, it raises a special number (roughly 2.718, written e) to the power of each score. This makes every value positive and stretches the gaps so a clear winner pulls ahead. Second, it divides each result by the total, so the whole set neatly sums to 1. If you like the formula, softmax for one score is: e^(that score) / (sum of e^(every score)). But the intuition is all you need: big scores become big probabilities, everything stays positive, and it all adds up to 100%.

Why softmax matters more than you’d think

This little function is quietly one of the most important pieces in modern AI, and here’s the payoff that ties into everything ahead: language models use softmax to choose the next word.

When a model like the one behind a chatbot is writing, its final step produces a raw score for every possible next token, tens of thousands of them. Softmax turns that giant list of scores into a giant list of probabilities: maybe “Paris” gets 70%, “France” gets 10%, and thousands of other options split the rest. The model then picks from that distribution. We’ll come back to this when we talk about how language models generate text, but the machinery is exactly the softmax you just met.

The two ingredients, in one breath

So here’s the whole article in a sentence: activation functions add the bends that let a network learn complex patterns, and softmax turns raw scores into clean probabilities so the network can make a confident choice.

We now have neurons that can bend, arranged in layers, capped off with softmax to produce decisions. But we’ve been quietly dodging the biggest question this whole time: how does the network actually learn the right weights? We keep saying “training adjusts the knobs,” but how does it know which way to turn each of millions of dials? That process is next.

---

Next in the series: How Networks Learn: Loss, Gradient Descent & Backpropagation: the process that actually tunes all those weights.