Data Generation
The PedalCity app team changed the rides table last week. Nobody told the data team, and this morning's dashboard is empty. Data engineers rarely control the systems that create data. Those systems were built to rent bikes, not to feed reports, so you have to study them the way a guest studies a house before rearranging the furniture.
The rule
You don't own the source — learn its shape, rhythm and quirks.
- Shape: what one record looks like and what it means. One row in
ridesis one rental. One message from a dock sensor is one reading of how many bikes are present. Know the columns, types and keys. - Rhythm: when and how fast data appears. Rides peak between 08:00 and 09:00. Each of 2,400 docks reports every 30 seconds, which comes to nearly 7 million readings a day.
- Quirks: the behaviour that surprises you. Does the source update rows or only add them? Does it delete? Are clocks reliable? Can columns change without warning?
PedalCity's rides table has a classic quirk. A row is inserted when the bike is unlocked, with ended_at empty, and updated when the bike is docked:
-- at unlock
INSERT INTO rides (ride_id, rider_id, start_station, started_at)
VALUES (88121, 5521, 17, '2026-09-24 07:52:10');
-- at dock, 22 minutes later
UPDATE rides
SET end_station = 4, ended_at = '2026-09-24 08:14:03', fare = 45.00,
updated_at = '2026-09-24 08:14:03'
WHERE ride_id = 88121;A copy taken mid-ride sees a half-finished row. A pipeline that only looks for new rows would never see the update.
Because you do not own the source, agree a data contract with its owners: a written promise about the schema, the meaning of fields and how much notice you get before changes.
The shortcut
Interview every new source with three questions: What does one record mean? How and how often does it change? Who tells me when its structure changes?
What does one record mean?
One row is one rental, identified by
ride_id. It is not one payment and not one trip leg.
Drills
Answer fast — trust the rule.
A ride row is inserted at unlock with ended_at empty and updated at docking. What does a copy taken during busy hours contain?
The app team renames fare to fare_inr without telling anyone. Which kind of quirk has bitten you?
Which interview question tells you what a record means?
What is a data contract?
Remember
- You don't own the source: learn its shape, rhythm and quirks.
- Ask three questions: what one record means, how it changes, who announces changes.
- Rows that are updated in place hide changes from pipelines that only look for new rows.