Header image

“Neural network” is one of those phrases that sounds far more intimidating than the thing it describes. It conjures images of a digital brain, something mysterious and vaguely alive. So let me pull the curtain back right away: a neural network is, at its heart, a large collection of very simple little calculators, wired together. That’s it. Understand one of those little calculators, and you understand the basic structure, because the network is many copies of it arranged in layers.

This is the machine that learns the embeddings from the last article, recognises faces, and, scaled up enormously, powers the language models this series is building toward. Let’s meet its building block.

The neuron: a tiny decision-maker

The basic unit is called a neuron (borrowed loosely from the cells in your brain, though the resemblance is loose). A single neuron does something almost embarrassingly simple: it takes in some numbers, decides how much each one matters, adds them up, and passes along a result.

Here’s the everyday version. Imagine you’re deciding whether to go for a walk. A few things feed into that decision:

  • Is the weather nice? (matters a lot)
  • Do I have free time? (matters a lot)
  • Is there a good podcast to listen to? (matters a little)

You weigh each factor by how much it counts, add up the evidence, and if the total crosses some threshold, you go. A neuron does exactly this with numbers. Figure 1 shows one.

The anatomy of a single neuron Figure 1: A neuron takes several inputs, multiplies each by a weight (how much it matters), adds them together along with a bias (a baseline nudge), and produces one output.

The two starring roles in Figure 1 are:

  • Weights: one per input, capturing how much that input matters. A big weight means “pay close attention to this”; a weight near zero means “basically ignore this.” The weather might get a big weight; the podcast a small one.
  • Bias: a single extra number that nudges the result up or down regardless of the inputs. Think of it as the neuron’s baseline mood: an eager neuron leans toward firing, a reluctant one holds back.

In plain terms, a neuron computes a weighted sum:

output = (weight1 × input1) + (weight2 × input2) + (weight3 × input3) + bias

That’s the entire calculation. Here it is in code, for our walk decision:

# Inputs: nice weather? free time? good podcast?  (1 = yes, 0 = no)
inputs  = [1, 1, 0]

# Weights: how much each factor matters
weights = [0.6, 0.5, 0.1]
bias    = -0.4      # a slightly reluctant baseline

total = sum(w * x for w, x in zip(weights, inputs)) + bias
print(round(total, 2))   # 0.7  -> comfortably positive, so: go for the walk

Weather (0.6) and free time (0.5) both fire, the podcast doesn’t matter, and even with a reluctant bias the total lands comfortably positive. The neuron “decides” to walk.

Weights are where the learning lives

Now for the most important sentence in this article: the weights and biases are exactly what a neural network learns.

When we said back in Article 2 that a model is “trained” by adjusting knobs until it fits the examples, these are the knobs. A freshly built network starts with random weights and makes nonsense predictions. Training is the process of nudging every weight and bias, over and over, until the network’s outputs match the examples we’re showing it. A big network has millions or billions of these knobs, and learning is the patient tuning of all of them.

So when you hear that a model “has 70 billion parameters,” those parameters are essentially the weights and biases across a vast web of neurons. Each one is a tiny dial the machine turned during training.

From one neuron to a network: layers

A single neuron can only make simple decisions. The magic comes from wiring lots of them together into layers, where the outputs of one layer become the inputs to the next. Figure 2 shows the classic arrangement.

Neurons arranged in layers Figure 2: Information flows left to right. The input layer receives the raw numbers, one or more hidden layers transform them step by step, and the output layer produces the final answer.

Figure 2 has three kinds of layers:

  • The input layer is just the numbers we feed in: the pixels of an image, or the embedding of a word.
  • The hidden layers are the workhorses in the middle. Each neuron combines what the previous layer produced, and layer by layer the network builds up more and more sophisticated notions. In an image network, early hidden layers might detect simple edges, middle layers might assemble those into shapes like eyes and wheels, and later layers might recognise whole concepts like “face” or “car”. Nobody told them to; they learned to. This automatic building-up of complexity is what the “deep” in deep learning refers to: simply having many hidden layers stacked up.
  • The output layer produces the final answer: a prediction, a category, a score.

The point is that information flows in one direction, each layer handing its work to the next, transforming raw numbers into a useful answer through a chain of simple weighted sums. No single neuron is clever. The cleverness lives in the arrangement and in the learned weights.

Why this simple idea is so powerful

Here’s what took me a while to appreciate. Every neuron is doing simple arithmetic: a weighted sum, a nudge. Yet stack enough of them, wire them into enough layers, and tune the weights on enough examples, and this simple structure can learn to recognise speech, translate languages, drive the embeddings from the last article, and, as we’ll build toward, model human language itself.

It works because those layers of weighted sums, tuned on data, can approximate complex patterns. You don’t program the intelligence in; you provide the structure and the examples, and the network finds the weights that do the job.

Two things are still missing from our picture, though, and the next couple of articles supply them. First, a plain weighted sum is a bit too simple on its own. Networks need a small extra ingredient to handle complicated patterns, and to turn their outputs into clean choices. And second, we’ve talked about learning the weights without saying how that tuning actually happens. Those two pieces, the extra ingredient and the learning process, are where we head next.

Next in the series: Activation Functions & Softmax: the small ingredient that lets networks handle real complexity and make clean decisions.