Spread and Outliers
Two branches can both average 3 copies a sale: one sells 3 every time, the other swings between 0 and 6. Same centre, very different shop to plan stock for. Spread measures how far values wander from the centre, and it is also how you spot values that wander too far.
The rule
Spread measures the wander; outliers sit beyond the fences.
Variance is the average squared distance from the mean. Standard deviation is its square root, which brings the answer back to the original units (copies, not copies squared).
sales["copies"].var() # about 2.79
sales["copies"].std() # about 1.67 copiespandas divides by n − 1 (the sample version). NumPy's np.var divides by n unless you pass ddof=1, so the two can disagree on the same numbers: np.var gives 2.4375 here.
The interquartile range (IQR) measures the spread of the middle half. Q1 is the value a quarter of the way up the sorted data, Q3 three quarters up:
q1 = sales["copies"].quantile(0.25) # 2.0
q3 = sales["copies"].quantile(0.75) # 4.25
iqr = q3 - q1 # 2.25Two common ways to flag outliers:
- IQR fences: anything below
Q1 − 1.5 × IQRor aboveQ3 + 1.5 × IQR. Here the fences are −1.375 and 7.625, so no sale is flagged. - Z-score:
(value − mean) / std, the number of standard deviations from the mean. Values beyond 3 (or −3) are flagged.
z = (sales["copies"] - sales["copies"].mean()) / sales["copies"].std()
outliers = sales[(sales["copies"] < q1 - 1.5 * iqr) | (sales["copies"] > q3 + 1.5 * iqr)]And always look: a box plot (coming soon) marks values beyond these fences as dots.
The shortcut
Prefer the IQR fences on small or skewed data. An extreme value inflates the mean and std it is measured against, so it can hide from its own z-score. Quartiles barely notice it.
Add the bulk sale
Copies become
3, 5, 2, 4, 1, 6, 2, 3, 60.
The trap is treating "outlier" as "error". Some are typos (600 instead of 6); some are your most important customers. Investigate before you drop.
Drills
Answer fast — trust the rule.
Which measure is in the same units as the data itself?
Q1 is 10 and Q3 is 18. What is the upper IQR fence?
A value is 2 standard deviations below the mean. What is its z-score?
Why did the 60-copy sale escape the z-score test?
Remember
- Spread measures the wander; outliers sit beyond the fences.
- Standard deviation is in the data's units; pandas uses
n − 1, NumPy usesnby default. - Prefer IQR fences on small or skewed data, and investigate outliers before dropping them.