Hypothesis Tests and p-values

A sceptical finance manager looks at the Quick Checkout result and says: "Conversion jumps around every week. Maybe the new checkout does nothing and you just caught a good fortnight." That is a fair challenge, and a hypothesis test is the formal way to answer it.

The rule

A p-value asks: if nothing changed, how surprising is data this extreme?

A test starts by assuming the boring explanation is true.

• The null hypothesis says the change has no effect: both checkouts convert at the same rate, and any gap is sampling noise.
• The test statistic measures how far the result sits from "no effect", in standard errors:

For the checkout test, $z = 1 / 0.36 \approx 2.8$. The observed lift is almost three standard errors away from zero.

• The p-value is the chance, if the null were true, of seeing a gap at least this far from zero in either direction. For $z \approx 2.8$ it is about 0.006: pure noise would produce a gap this large only about 6 times in 1,000 tests.
• Before looking, you choose a significance level, usually $\alpha = 0.05$. If the p-value is below it, you call the result statistically significant and reject the null.

A p-value is easy to over-read. It is not the probability that the null is true, and it is not the size of the effect. With millions of users, a lift too small to matter can still earn a tiny p-value, so always put the effect and its interval next to it. Tests and intervals agree: a 95% interval that excludes zero goes with a two-sided p-value below 0.05.

The shortcut

Divide the difference by its standard error. Beyond about 2, in either direction, is surprising at the 5% level; beyond about 2.6 is surprising at 1%. You can often sort a result in your head before any software runs.

Absence of evidence is not evidence of absence. A non-significant result means the data could not tell the change apart from noise, which is a statement about the test as much as about the shelf.

Drills

Answer fast — trust the rule.

Remember

• A p-value asks how surprising the data would be if nothing had changed.
• z = difference ÷ SE; beyond about ±2 is significant at 5%.
• Trap: reading p as the chance the null is true, or treating "not significant" as "no effect".