From Decision to Metric
Pantry is a grocery-delivery app in Pune, and it is the company you will work for throughout this course. On your first morning the head of product sends one line: "Is the new checkout better?" You could open a notebook and start plotting, but "better" is not a number yet, and an analysis aimed at a vague word can only return a vague answer.
The rule
Name the decision first, then choose the number that would change it.
A question becomes answerable when you can fill in four blanks:
- Decision: what will someone do differently depending on the answer? Here, Pantry will either switch every user to the new one-page "Quick Checkout" or keep the old three-step flow.
- Metric: a ratio with a clear numerator (what you count) and denominator (what you count it out of). Users who complete an order, divided by users who open checkout.
- Unit: who or what one observation is. A user, a session or an order give different answers, because one keen shopper can open checkout forty times in a week.
- Threshold: the value that flips the decision, agreed before anyone sees the data.
Written down, it becomes a short brief that everyone can sign off:
decision: switch all users to Quick Checkout, or keep the old flow
metric: users who complete an order / users who open checkout
unit: user
window: 7 days after a user first opens checkout
threshold: switch if conversion rises by at least 0.5 points
guardrail: refunds per order must not riseThe guardrail is a second number that must not get worse. A checkout that sells more but causes more refunds is not better.
The shortcut
Say "decision, numerator, denominator, threshold" in that order, every time. If any one is missing, you are not ready to query.
Decision
Ask what hangs on it. The operations head is deciding whether to hire extra riders for the evening rush.
The shortcut also catches the most common trap: a count with no denominator. Late evening deliveries rose from 300 to 360 last month, which sounds worse. But evening orders rose from 2,000 to 3,000, so the late share actually fell from 15% to 12%.
Drills
Answer fast — trust the rule.
A manager asks you to "check whether the new search bar is better." What should you ask first?
Which metric fits the decision "should we hire extra evening riders?"
Why agree the threshold before looking at the results?
One shopper opens checkout 40 times a week. Which unit stops heavy users from dominating the checkout metric?
Remember
- Decision first; the metric is whatever number would change that decision.
- Decision, numerator, denominator, threshold, and write down the unit.
- Trap: a rising count with no denominator. Always ask "out of how many?"