Fixing Types and Parsing Dates
The owner asks, "Do we sell more at weekends?" But the date column is plain text like "2024-03-02", and text has no idea what a Saturday is. Worse, a copies column loaded as text will "add" 3 and 5 into "35". Every column needs the type that matches its meaning before the analysis can be trusted.
The rule
Give each column the type of its meaning: numbers, dates, or categories.
Numbers. astype converts when every value is clean; to_numeric copes when some are not:
s = pd.Series(["3", "5", "2"])
s.sum() # '352', text glued together
s.astype(int).sum() # 10
bad = pd.Series(["3", "5", "two"])
bad.astype(int) # ValueError
pd.to_numeric(bad, errors="coerce") # 3.0, 5.0, NaNerrors="coerce" turns anything unreadable into NaN, which you then handle like any other missing value.
Dates. pd.to_datetime turns text into real timestamps, and the .dt accessor unlocks their parts:
sales["date"] = pd.to_datetime(sales["date"])
sales["date"].dt.day_name() # Friday, Friday, Saturday, Saturday, Sunday, …
sales["date"].dt.month # 3 for every row
sales["date"].max() - sales["date"].min() # 3 daysAmbiguous text is a real danger: "03/04/2024" is read month first, as 4 March. For Indian-style day-first dates, pass dayfirst=True to get 3 April, or better, give an exact format="%d/%m/%Y".
Categories. A text column with a few repeating values, like genre, can become the category type, which stores each label once and saves memory:
sales["genre"] = sales["genre"].astype("category")
sales["genre"].cat.categories # comics, fiction, science
pd.get_dummies(sales["genre"]) # one True/False column per genreget_dummies is how categories are turned into numbers for models: three genres become three yes/no columns.
The shortcut
Read dtypes and ask of each column: "Does this type let me do what I will ask of it?" Adding needs a number type. Weekdays, months and gaps between days need datetime64. A handful of repeating labels suits category.
Check the type
sales["date"].dtypeisobject: text. It cannot answer weekday questions.
The trap is a column that looks numeric in head() but has dtype object. Sorting it gives "10" before "9", and sums glue strings. Trust dtypes, not your eyes.
Drills
Answer fast — trust the rule.
A column holds the text values "4", "1", "6". What does .sum() return?
What does pd.to_numeric(pd.Series(["7", "n.a."]), errors="coerce") produce?
How does pd.to_datetime("05/06/2024") read the date by default?
How many columns does pd.get_dummies(sales["genre"]) create?
Remember
- Give each column the type of its meaning: numbers, dates, or categories.
to_numeric(errors="coerce")andto_datetimerescue messy text;.dtunlocks date parts.- Slash dates default to month first: use
dayfirst=Trueor an exactformat.