Finding Missing Values
Last night's export arrived with holes: the till crashed twice and one clerk skipped a field. Any average you compute now is quietly averaging over gaps. Before fixing missing values, you have to find them all and know where they sit.
The rule
Ask
isna, thensum: missing values counted per column.
Here is the raw export, loaded as raw:
| date | branch | genre | copies | price | |
|---|---|---|---|---|---|
| 0 | 2024-03-01 | Pune | fiction | 3.0 | 250.0 |
| 1 | 2024-03-01 | Delhi | science | 5.0 | 400.0 |
| 2 | 2024-03-02 | Pune | science | NaN | 400.0 |
| 3 | 2024-03-02 | Chennai | NaN | 4.0 | 250.0 |
| 4 | 2024-03-03 | Delhi | fiction | 1.0 | NaN |
| 5 | 2024-03-03 | Chennai | comics | 6.0 | 120.0 |
| 6 | 2024-03-04 | Pune | comics | NaN | 120.0 |
| 7 | 2024-03-04 | Delhi | science | 3.0 | 400.0 |
pandas marks a missing value as NaN ("not a number"); a missing None in a text column also counts as missing. Notice copies shows 3.0 rather than 3: a column of whole numbers with a NaN in it is stored as floats.
raw.isna() # same-shape table of True/False
raw.isna().sum() # date 0, branch 0, genre 1, copies 2, price 1
raw.isna().sum().sum() # 4 missing cells in total
raw.isna().any(axis=1) # True for each row with at least one gap
raw[raw.isna().any(axis=1)] # show those rows: 2, 3, 4 and 6isnull() is exactly the same method under an older name; notna() is the opposite. raw.info() also shows the non-missing count per column, a quick way to spot gaps.
Why a special method? Because NaN is not equal to anything, not even itself:
import numpy as np
np.nan == np.nan # FalseSo raw["copies"] == np.nan is False on every row and finds nothing. Always use isna().
The shortcut
Run isna().sum() right after the three-glance check, and read it as a short report: which columns have gaps, and how many. Then look at the rows themselves with the any(axis=1) mask, because the fix depends on why the gaps are there.
Count per column
raw.isna().sum()reports genre 1, copies 2, price 1.
The trap is trusting a summary that hides gaps. raw["copies"].mean() happily returns a number: pandas skips NaN by default, so it averages the six known values without telling you two are missing.
Drills
Answer fast — trust the rule.
What is raw["copies"].isna().sum()?
What does np.nan == np.nan evaluate to?
What is the difference between isna() and isnull()?
What does raw["copies"].mean() return?
Remember
- Ask
isna, thensum: missing values counted per column. - Use
any(axis=1)to find and inspect the incomplete rows. - NaN never equals NaN, and summaries skip it silently.