Derivatives and Partial Derivatives

Training a model is a long series of small questions: "If I nudge this weight up a little, does the error go up or down, and by how much?" The derivative answers that question. With several weights, the partial derivative answers it for one weight at a time.

The rule

Derivative: output change per tiny input change. Partial: freeze every other input.

A derivative is the slope of a function at a point. Take $f(w) = w^2$. Its derivative is $f'(w) = 2w$, so at $w = 3$ the slope is 6. That means a tiny nudge in $w$ changes $f$ about six times as much: $f(3) = 9$ and $f(3.01) = 9.0601$, a rise of about $6 \times 0.01$.

The rules you need most:

• Power rule: the derivative of $w^n$ is $n w^{n-1}$. So $5w^3$ gives $15w^2$.
• Constants vanish: a number on its own has slope 0.
• Constant multiples stay: the derivative of $3w$ is 3.
• Sums split: differentiate each term separately.

The sign matters as much as the size. A positive slope means increasing $w$ increases $f$. A negative slope means increasing $w$ decreases $f$. To reduce an error, move the weight against the slope.

A model usually has many weights. A partial derivative measures the slope along one of them while every other weight is treated as a fixed number. For $f(w, b) = w^2 + 3wb$:

• $\partial f / \partial w = 2w + 3b$, because $b$ is frozen, so $3wb$ is just $3b$ times $w$.
• $\partial f / \partial b = 3w$, because $w^2$ has no $b$ in it and vanishes.

At $(w, b) = (1, 2)$ these are 8 and 3.

The shortcut

Freeze, then power-rule. Before differentiating with respect to one variable, mentally replace every other variable with a number. What is left is a one-variable problem you already know how to solve.

The usual slip is forgetting a cross term like $2wb$: it has a slope in both directions.

Drills

Answer fast — trust the rule.

Remember

• Derivative: output change per tiny input change. Partial: freeze every other input.
• Shortcut: freeze the others, then use the power rule.
• Trap: dropping cross terms like 2wb, which have a slope in both directions.