The Lifecycle at a Glance
A PedalCity ride lasts twenty minutes, but its data has a much longer life. It is created by the app, copied, stored, cleaned, joined, summed and finally shown on a dashboard. When a number on that dashboard looks wrong, you need a map of that journey to know where to look. The data engineering lifecycle is that map.
The rule
Generate → ingest → transform → serve, with storage underneath every step.
The four stages happen in order:
- Generation: a source system creates the data. A rider docks a bike and the app writes a row to its
ridestable. - Ingestion: the data is moved from the source into your platform. A nightly job copies yesterday's new rides into cloud storage.
- Transformation: the data is cleaned and reshaped. Cancelled rides are dropped, UTC times become local times, and station names are attached.
- Serving: the data reaches the people and programs that use it. The ops dashboard, the finance team and the forecast model all read the result.
Storage is not a fifth step at the end. It sits under all of them: the source stores its own data, ingested files are stored in raw form, and transformed tables are stored again for serving. Choosing how and where to store things is a decision you make at every stage.
Some concerns run beneath the whole lifecycle too, like currents under a river: security (who may see rider emails), data quality (are the counts right), orchestration (what runs when) and cost. They are not stages. They are checks you apply at every stage.
The shortcut
Place any task by asking: "Which stage does this touch?" Once you know the stage, you know what can go wrong there. Generation breaks through source changes. Ingestion breaks through missing or duplicated copies. Transformation breaks through wrong logic. Serving breaks through stale or misunderstood tables.
Generation
At 08:14 rider 5521 docks bike 903 at station 17. The app updates the ride row with an end time and a fare.
When the dashboard shows zero rides for station 17, walk the stages backwards. Did the source record rides, did the copy arrive, did a filter remove them, or is the dashboard reading an old table?
Drills
Answer fast — trust the rule.
A rider docks a bike and the app writes the ride's end time to its database. Which stage is this?
A nightly job copies new rows from the app database into cloud storage. Which stage?
Converting timestamps to local time and attaching station names happens in which stage?
Why does storage sit underneath the lifecycle rather than being one step?
Remember
- Generate → ingest → transform → serve, with storage under every step.
- Security, quality, orchestration and cost apply to every stage.
- Debug a wrong number by walking the stages backwards.