A/B Testing and Feature Flags
The morning change window is ready, but one argument remains. The cooks' representatives want it to close at 7:00 a.m., for a safer buffer. The design team wants 7:30 a.m., which catches more late changes. The last lesson showed that comparing people who chose to use a feature cannot prove cause. So LunchLoop lets subscribers settle it through their behaviour, with a switch ready to undo everything if something breaks.
The rule
Fix the decision rule before the test; let the flag decide who sees what.
An A/B test splits people at random into groups, shows each group one version, and compares a metric chosen in advance. Random assignment makes the groups alike in every other way, so a difference in results can be credited to the change itself. That is what turns correlation into cause.
The rules that keep a test honest:
- Decide before you start: one primary metric, the guardrails, and the size of difference that would change your mind.
- Run long enough: work out the sample size you need in advance, and run in whole weeks so that every weekday is covered.
- No stopping at the first good-looking result. Checking again and again and stopping the moment one version leads produces false wins.
- Assign by person, not by visit, so each subscriber sees the same version throughout.
A feature flag is a switch in the code that turns a feature on or off for chosen people without shipping a new release. Flags let a team split traffic for a test, release gradually (5%, then 25%, then everyone), let staff try things first, and switch a feature off instantly if it misbehaves.
if (flags.isOn("change_window_730", subscriber.id)) {
showChangeUntil("7:30 a.m.");
} else {
showChangeUntil("7:00 a.m.");
}Old flags are removed once a decision is final; forgotten flags turn into tangled code.
The shortcut
Write the decision before the data. "If B cuts untouched boxes by at least 1 percentage point and cook leftovers do not rise, we ship B; otherwise we keep A." If you cannot write that sentence, you are not ready to test.
Write the decision rule
Primary metric: untouched-box rate. Guardrail: leftover portions per cook per day. Ship 7:30 if it cuts untouched boxes by 1 point or more while leftovers do not rise.
Drills
Answer fast — trust the rule.
Why must people be assigned to the A and B groups at random?
Two days into a planned three-week test, version B looks 20% better. What should the team do?
What does a feature flag let a team do?
B cuts untouched boxes by 2 points, but cook leftovers double. Under the pre-written rule, what happens?
Remember
- Fix the decision rule before the test; let the flag decide who sees what.
- Shortcut: write "if the result is X, we do Y" before any data arrives.
- Trap: stopping at the first good-looking result, and leaving old flags in the code.