Pebellslearn by building
0%
RoadmapsMatrixCoursesSandboxSign in
Statistical Inference  ›  Capstone Project

Errors, Power and Sample Size

Statistics 15 minA test can raise a false alarm or miss a real effect, and choosing the smallest effect worth detecting before the test fixes how many users it needs to have enough power.

Errors, Power and Sample Size

Before the checkout test, someone else at Pantry tested a "Reorder" button for three days with 1,000 users per arm. The result was p = 0.4, and the product manager declared: "The button doesn't work." Maybe. Or maybe the test was far too small to notice a button that does work. Deciding how big a test must be before it starts is what separates an experiment from a coin toss.

The rule

Decide the smallest effect worth finding, then buy enough sample to see it.

Every test can go wrong in two ways:

  • A Type I error is a false alarm: declaring a winner when the change does nothing. Its rate is the significance level, α\alpha, usually 5%.
  • A Type II error is a miss: failing to detect a change that really works. Power is the chance of not missing, and teams usually aim for 80%.

Power depends on the size of the effect you hope to catch. The minimum detectable effect (MDE) is the smallest lift worth acting on, and it comes from the decision, not from the data. For comparing two conversion rates at α=0.05\alpha = 0.05 with 80% power, a well-known rule of thumb is

nper arm≈16 p(1−p)δ2n_{\text{per arm}} \approx \frac{16\, p(1-p)}{\delta^2}

where pp is the baseline rate and δ\delta is the MDE, both as proportions. For Quick Checkout, p=0.1p = 0.1 and δ=0.01\delta = 0.01 (one point): 16×0.09/0.0001=14,40016 \times 0.09 / 0.0001 = 14{,}400 users per arm. That is exactly why the test ran with 14,400 users in each group.

Because δ\delta is squared, halving the effect you want to detect needs four times the users.

The shortcut

Sixteen times p(1 − p), divided by the lift squared, then turn users into days.

Sizing the Reorder button test properlyStep 1 of 4

  1. Baseline and MDE

    20% of users reorder within 14 days. A lift of 2 points would pay for the feature, so p = 0.2 and the MDE is 0.02.

The three-day test had 1,000 users per arm, less than a sixth of what this lift needs. Its p = 0.4 says "too little data to tell", not "no effect". Small tests have a second problem: when they do come out significant, they tend to exaggerate the lift, because only a lucky overestimate clears the bar.

Drills

Answer fast — trust the rule.

quick drill

A test declares the new checkout a winner, but in truth it makes no difference. Which error is that?

Using the rule of thumb, how many users per arm are needed to detect a 1-point lift from a 5% baseline?
16×0.05×0.95÷0.012= ?16 \times 0.05 \times 0.95 \div 0.01^{2} = \,?
quick drill

You shrink the MDE from 1 point to 0.5 points. The sample you need:

quick drill

The three-day Reorder test had 1,000 users per arm and p = 0.4. What is the best reading?

A test needs 6,400 users per arm, with two arms, and 1,000 users join per day. How many days of traffic is that?
2×64001000= ?2 \times \tfrac{6400}{1000} = \,?

Remember

  • Choose the smallest effect worth finding first, then size the test to see it.
  • n per arm ≈ 16 p(1 − p) ÷ MDE², and halving the MDE needs four times the users.
  • Trap: calling an underpowered, non-significant test proof that nothing works.

Practice — level up

0/1 solved

Write your answer, then compare it with a rubric and a model answer.

ExplainKeep the test running

Three days into a properly sized Reorder-button test, the product manager wants to stop because the result is not significant yet. The plan needs 6,400 users per arm over 14 days. Explain power and the minimum detectable effect in plain words, and why stopping now would be a mistake.