Finding Outliers

Saturday's 95 cold coffees sit far above every other day that week. Is it a typo, a till glitch, or the busiest day of the year? An outlier can be the most important number in the dataset or the most misleading one. The job is to find it with a fair rule, then decide what it is before deciding what to do.

The rule

Flag with a rule, investigate with context, then decide.

The standard rule is the 1.5 × IQR fence:

1. Sort the data and find the quartiles. Q1 is the middle of the lower half; Q3 is the middle of the upper half.
2. The interquartile range is IQR = Q3 − Q1, the width of the middle half of the data.
3. The fences are Q1 − 1.5 × IQR and Q3 + 1.5 × IQR. Anything outside is flagged.

For the week, sorted: 36, 38, 38, 40, 43, 46, 95. The median is 40. Leaving it out, the lower half is 36, 38, 38, so Q1 = 38; the upper half is 43, 46, 95, so Q3 = 46.

• IQR = 46 − 38 = 8
• Lower fence = 38 − 12 = 26
• Upper fence = 46 + 12 = 58

95 is above 58: flagged. (Spreadsheets and software use slightly different quartile methods, so their fences can differ a little from a hand calculation.)

The rule works because quartiles barely move when one value is extreme. A rule based on the mean and standard deviation, such as "more than 3 standard deviations from the mean", is weaker on small data: the outlier inflates the standard deviation it is measured against. Here 95 is only (95 − 48) ÷ 21 ≈ 2.2 standard deviations out, so that rule would miss it.

The shortcut

Once a value is flagged, sort it into one of three bins: error, event, or extreme-but-normal.

• Error (a typo, a price of ₹12,000 for a coffee): fix it from the source, or remove it and log why.
• Event: keep it, label it, and handle it separately where it would distort a "normal" picture.
• Extreme but normal (a big catering order that happens now and then): keep it. It is part of how the business works.

The trap is deleting outliers automatically. They are sometimes the most interesting thing in the data.

Drills

Answer fast — trust the rule.

Q1 is 38 and Q3 is 46. What is the upper fence?

Remember

• Flag with a rule, investigate with context, then decide.
• Shortcut: fences at Q1 − 1.5 × IQR and Q3 + 1.5 × IQR; then error, event, or extreme-but-normal.
• Trap: deleting outliers automatically.