Designing an A/B Test
To find out whether Quick Checkout causes more orders, Pantry cannot simply compare users who happened to see it with users who did not; that invites every rival from the last lesson. Instead, it lets chance decide. Each user is assigned to the old checkout (A) or the new one (B) by what is effectively a coin flip, and everything else is written down before the first user arrives.
The rule
Let chance assign the change, so the change is the only difference.
When a coin decides who gets B, keen cooks, loyal users, rich neighbourhoods and every factor nobody thought to record end up split evenly between the arms, on average. So if B converts better, the only systematic explanation left is B itself. That is why a randomised experiment can do what no amount of observational data can.
Randomisation only protects you if the rest of the design is fixed in advance. A good test plan settles:
- Unit of randomisation: who gets assigned. For a checkout, the user, so one person never sees both versions.
- Primary metric: one number that decides the test, straight from the question brief.
- Guardrail metrics: numbers that must not get worse, such as refunds or page-load time.
- Sample size and duration: from the power calculation, in whole weeks.
- Decision rule: what result leads to which action.
test: quick-checkout-v1
hypothesis: a one-page checkout raises checkout conversion
unit: user_id, hashed into A or B, split 50/50
primary metric: users who order / users who open checkout, within 7 days
guardrails: refund rate, checkout page-load time, support tickets per order
minimum detectable effect: +1 point from a 10% baseline
sample: 14,400 users per arm, about 2 weeks
decision: switch to B if the lift is significant at 5%, is estimated at 0.5 points or more, and no guardrail is harmedHashing the user ID means the same user lands in the same arm every visit, and nobody can choose their arm.
The shortcut
Before launch, fill in unit, metric, size, stop. If you cannot write all four in one breath, the test is not ready.
Unit
The user. The button appears on every visit, so assigning by visit would show it to the same person on and off.
Letting users opt in to the new version breaks everything the coin was for. Volunteers are keener, so their better numbers say nothing about the change.
Drills
Answer fast — trust the rule.
Why randomise Quick Checkout by user rather than by page view?
What does random assignment balance between the arms?
Which is a guardrail metric for the Quick Checkout test?
Pantry lets users opt in to try Quick Checkout, then compares them with everyone else. What is the problem?
Remember
- Let chance assign the change, so the change is the only difference.
- Fix unit, metric, size and stop before launch.
- Trap: opt-in groups, per-visit assignment and early stopping all quietly undo randomisation.