Reading an A/B Test Result

Two weeks after launch, the Quick Checkout dashboard fills with numbers: two conversion rates, a p-value, a dozen secondary metrics and a chart per city. It is tempting to jump straight to the biggest green number. Reading a test well means looking at things in a fixed order, because the first check can make every later number meaningless.

The rule

Check the split, then the effect's range, then the guardrails, then decide.

• The split comes first. If the plan was 50/50 and one arm has far more users than chance allows, something is broken: a bug in assignment, a crash that drops some users, or logging that misses one arm. This is called a sample ratio mismatch, and when you find one you stop and fix it; the metrics cannot be trusted.
• The effect's range comes next: the lift, its confidence interval and its p-value, all for the primary metric named in the plan.
• The guardrails: did anything that must not get worse, get worse?
• The decision follows the rule written before launch, not the mood in the room.

For a 50/50 test with $N$ users in total, each arm's count naturally drifts by up to about $\sqrt{N}$ from $N/2$. Here $\sqrt{28{,}800} \approx 170$, so anything within 170 of 14,400 is normal.

The shortcut

Read in the order split, primary, guardrails, rule, and write one line per step.

Two habits quietly ruin good tests:

• Peeking: checking the p-value every day and stopping the first time it dips below 0.05. Each look is another chance for noise to cross the line, so the real false-alarm rate climbs several times above 5%. Decide once, at the planned end.
• Fishing: scanning many secondary metrics or city-by-city slices and celebrating whichever one is significant. With enough looks, something always is. Treat those findings as ideas for the next test.

Drills

Answer fast — trust the rule.

Remember

• Split, primary, guardrails, rule: in that order, every time.
• Each arm should be within about √N of half the total; a bigger gap means a bug.
• Traps: peeking and fishing. Decide once, on the metric you named in advance.