Pebellslearn by building
0%
RoadmapsMatrixCoursesSandboxSign in
Calculus  ›  Capstone Project

Gradient, Jacobian and Hessian

Gradient, Jacobian, Hessian 16 minThe gradient lists the slopes of one output, the Jacobian lists the slopes of many outputs, and the Hessian lists the slopes of the slopes.

Gradient, Jacobian and Hessian

A real model has many weights and makes many predictions. Writing one partial derivative at a time quickly becomes a mess. Three named collections keep it tidy, and each one appears in ML papers and error messages all the time.

The rule

Gradient: slopes of one output. Jacobian: slopes of many outputs. Hessian: slopes of slopes.

Give the rent model a bias: prediction =6w+b= 6w + b for flat B, whose rent is 22. The loss is L(w,b)=(6w+b−22)2L(w, b) = (6w + b - 22)^2. Call the error e=6w+b−22e = 6w + b - 22.

Gradient: all first partials of a single number, stacked in a vector. By the chain rule,

∇L=(∂L∂w,∂L∂b)=(12e,  2e)\nabla L = \left( \frac{\partial L}{\partial w}, \frac{\partial L}{\partial b} \right) = (12e, \; 2e)

At (w,b)=(3,1)(w, b) = (3, 1): e=18+1−22=−3e = 18 + 1 - 22 = -3, so ∇L=(−36,−6)\nabla L = (-36, -6). The gradient points in the direction of steepest increase. Gradient descent steps the other way: new weights == old weights −- learning rate ×\times gradient. With learning rate 0.01, ww becomes 3+0.36=3.363 + 0.36 = 3.36 and bb becomes 1+0.06=1.061 + 0.06 = 1.06. The error shrinks to −0.78-0.78 and the loss drops from 9 to about 0.61.

Jacobian: when there are several outputs, stack each output's gradient as a row. Predictions for flat A (x=4x = 4) and flat B (x=6x = 6) are 4w+b4w + b and 6w+b6w + b, so

J=(4161)J = \begin{pmatrix} 4 & 1 \\ 6 & 1 \end{pmatrix}

Rows are outputs, columns are inputs.

Hessian: the partials of the partials of one number, showing how the slope itself changes (the curvature). For f=w2+3wb+2b2f = w^2 + 3wb + 2b^2, the gradient is (2w+3b,  3w+4b)(2w + 3b, \; 3w + 4b), and differentiating again gives

H=(2334)H = \begin{pmatrix} 2 & 3 \\ 3 & 4 \end{pmatrix}

It is symmetric, because the order of mixed partials does not matter for smooth functions.

The shortcut

Count inputs and outputs, and the shapes fall out. With nn inputs and mm outputs: the gradient has nn entries, the Jacobian is m×nm \times n, and the Hessian is n×nn \times n.

Shapes for the three-weight rent modelStep 1 of 4

  1. Count

    Three weights (size, rooms, metro) are the inputs. Five flats give five predictions.

Mixing up the Jacobian's orientation is the usual trap: outputs down the side, inputs along the top.

Drills

Answer fast — trust the rule.

quick drill

A loss depends on 3 weights. How many entries does its gradient have?

quick drill

What shape is the Jacobian of 5 predictions with respect to 3 weights?

quick drill

What is the Hessian of f = w² + 3wb + 2b²?

quick drill

Starting at (w, b) = (3, 1) with gradient (−36, −6) and learning rate 0.01, where does one gradient-descent step land?

quick drill

Why does gradient descent step against the gradient?

Remember

  • Gradient: slopes of one output. Jacobian: slopes of many outputs. Hessian: slopes of slopes.
  • Shortcut: n inputs, m outputs give n, m × n and n × n.
  • Trap: stepping along the gradient instead of against it, or flipping the Jacobian.

Practice — level up

0/1 solved

Write your answer, then compare it with a rubric and a model answer.

ExplainOne step downhill

For flat B the loss is L(w, b) = (6w + b - 22)². Starting at w = 3, b = 1 with learning rate 0.01, compute the gradient, take one gradient-descent step, and explain why the step goes against the gradient. Then say what shape the Jacobian of five predictions with respect to three weights would have.