Confidence Intervals

The Quick Checkout test has finished. The old checkout converted 10% of its 14,400 users and the new one converted 11% of its 14,400. "Conversion up 1 point" is true, but it hides the question the product lead actually cares about: could the real improvement be tiny, or even zero, and this was a lucky fortnight?

The rule

Report a range, not a point: estimate plus or minus two standard errors.

A 95% confidence interval is roughly

(the exact multiplier for 95% is 1.96; 2 is close enough to reason with). For a difference between two independent groups, the standard errors combine like the sides of a right-angled triangle:

For the checkout test, the old arm's SE is 0.25 points (from the last lesson) and the new arm's is about 0.26 points, so $SE_{\text{diff}} = \sqrt{0.25^2 + 0.26^2} \approx 0.36$ points. The interval is 1 ± 0.72: 0.28 to 1.72 points.

What "95%" means: the method that builds these intervals catches the true value in about 95 out of every 100 samples you could have drawn. It is a promise about the procedure, so the safe plain-language reading is "the data fit a lift anywhere from about 0.3 to 1.7 points."

The width comes straight from the standard error. More data, or less variable data, gives a narrower interval. Asking for more confidence, such as 99%, gives a wider one.

The shortcut

Build it with plus-or-minus-two, then read both ends against the decision. The estimate in the middle is the least interesting number.

The trap is reporting only the middle. "Up 1 point" and "up somewhere between 0.28 and 1.72 points" lead to different conversations, and the second one is the truth.

Drills

Answer fast — trust the rule.

Remember

• Report a range: estimate ± 2 standard errors.
• Read both ends: is zero inside, and does the low end still meet the decision threshold?
• Trap: announcing only the middle number.