Simpson's Paradox: Segment Before You Conclude
Before the proper A/B test, the web team switched Quick Checkout on for most web users, while the app team released it to only a small group. A month later someone compared the totals: the old checkout converted 17% of users and Quick Checkout only 9.2%. "The new checkout is a disaster," read the email. The A/B test nearly never happened. Yet on the app and on the web, taken separately, Quick Checkout was doing better.
The rule
When groups mix differently, compare within each group before trusting the total.
| Platform | Old checkout | Quick Checkout |
|---|---|---|
| App | 800 of 4,000 = 20% | 220 of 1,000 = 22% |
| Web | 50 of 1,000 = 5% | 240 of 4,000 = 6% |
| Total | 850 of 5,000 = 17% | 460 of 5,000 = 9.2% |
Quick Checkout wins on the app (22% against 20%) and wins on the web (6% against 5%), yet loses overall. This reversal is Simpson's paradox, and there is nothing mysterious about it. Two facts combine:
- The segments differ a lot in their rates. App users convert around 20%, web users around 5%, whichever checkout they see.
- The two options got very different mixes. 80% of the old checkout's users were on the app; 80% of Quick Checkout's users were on the web.
So the totals mostly compare app users with web users, not one checkout with the other. A total is a weighted average of its segments, and here the weights were stacked against the new checkout.
The shortcut
Check the mix. For each option, ask what share of its users sat in each segment. If the mixes differ and the segments convert differently, the total can mislead, so compare within segments or give both options the same mix.
Read the totals
17% against 9.2%. Quick Checkout looks almost twice as bad.
Two cautions. Split by things that were fixed before the change, such as platform or city, not by things the change could itself affect, such as basket size after checkout. And do not slice into dozens of segments hunting for reversals; with enough slices, some will flip by chance.
Drills
Answer fast — trust the rule.
What made Quick Checkout's total look worse than the old checkout's?
Which design prevents this kind of reversal even for segments you never thought to check?
Which variable is safe to split by when judging Quick Checkout?
Remember
- When groups mix differently, compare within each group before trusting the total.
- Check the mix: different weights plus different segment rates can flip a total.
- Trap: splitting by variables the change itself affects, or slicing until something flips.