Dropping, Imputing and Filling Gaps
You found four missing cells in the raw export. Now you must decide what to do with each. Drop the row, fill in a sensible value, or copy a neighbour's value? Each choice changes your final numbers, so the choice should be deliberate, not a habit.
The rule
Drop what you cannot trust, fill what you can reason about.
Three tools, using raw from the last lesson (gaps: copies in rows 2 and 6, genre in row 3, price in row 4).
Drop removes rows (or columns) with gaps:
raw.dropna() # any gap: 4 rows left
raw.dropna(subset=["copies"]) # only if copies is missing: 6 rows left
raw.dropna(axis=1) # drop columns with any gap insteadImpute replaces a gap with a chosen value, often a summary of the column:
raw["copies"].fillna(raw["copies"].median()) # median of known values is 3.5
raw["genre"].fillna("unknown") # a clear label for textForward / backward fill copies a neighbour. This fits ordered data, like a daily reading that carries on until it changes:
raw["copies"].ffill() # row 2 takes row 1's 5; row 6 takes row 5's 6
raw["copies"].bfill() # row 2 takes row 3's 4; row 6 takes row 7's 3Each method returns a new Series or DataFrame; assign it back to keep it: raw["genre"] = raw["genre"].fillna("unknown").
How to choose:
- Drop when few rows are affected and they are missing at random, or the missing field is essential (a sale with no price cannot count toward revenue).
- Impute with the median for skewed numbers, the mean for balanced ones, and an explicit label for categories.
- Fill forward/backward only when the order of rows means something, such as time.
The shortcut
For each column with gaps, ask "What would I tell my manager this value is?" If you cannot say anything honest, drop it. If you can ("probably typical", "same as yesterday", "genre unknown"), fill it with exactly that.
price, row 4
A sale with no price cannot give revenue, and guessing a price invents money. Drop:
raw.dropna(subset=["price"]).
The trap is ffill on rows whose order is meaningless. Here it would give Pune's row 2 the value from a Delhi sale just because they sit next to each other.
Drills
Answer fast — trust the rule.
How many rows does raw.dropna() keep?
After raw["copies"].ffill(), what is the value in row 6?
After raw["copies"].bfill(), what is the value in row 2?
Order values are mostly small, with a few huge bulk orders. Which fill is safest for a missing value?
Remember
- Drop what you cannot trust, fill what you can reason about.
- Median for skewed numbers, a clear label for categories, ffill/bfill only for ordered data.
- Every fill is an estimate: write down what you changed.