The Chain Rule

A model's error is never a simple function of a weight. The weight makes a prediction, the prediction makes an error, and the error gets squared. That is a chain of functions, and the chain rule is how you differentiate through it. Backpropagation, the method that trains every neural network, is this one rule used over and over.

The rule

Outside derivative times inside derivative; multiply along the chain.

Use the simplest rent model: predicted rent $= w \times x$, where $x$ is size in tens of square metres. For flat B, $x = 6$ and the true rent is 22, so the squared-error loss is

Break it into links:

• Inside: $u = 6w - 22$, the prediction error. Its slope is $du/dw = 6$.
• Outside: $L = u^2$. Its slope is $dL/du = 2u$.

Multiply: $dL/dw = 2u \times 6 = 12(6w - 22)$.

At $w = 3$: the prediction is 18, so $u = -4$ and $dL/dw = 12 \times (-4) = -48$. The slope is negative, so increasing $w$ lowers the loss. That makes sense: 18 is below 22, so the weight is too small.

Longer chains work the same way. A neural network might compute $z = wx + b$, then $s = \sigma(z)$, then $L = (s - y)^2$. Then $dL/dw = 2(s - y) \times \sigma'(z) \times x$: one factor per link. Backpropagation computes these factors from the loss backwards, layer by layer, and reuses each product for the layer below, which is why it is fast.

The shortcut

Name each link, write its local slope, multiply. Each link only needs its own small derivative; you never differentiate the whole expression at once.

The classic mistake is stopping at the outside derivative and writing $2(6w - 22)$. That forgets the inside slope of 6, and the answer is six times too small.

Drills

Answer fast — trust the rule.

Remember

• Outside derivative times inside derivative; multiply along the chain.
• Shortcut: name each link, write its local slope, multiply.
• Trap: forgetting the inside slope.