Data Lineage
On the morning of 3 April, Farah messages Ishaan: "On-time for yesterday says 70%. Is Kolkata on fire?" It usually sits around 90%. Either something terrible happened in the warehouses, or something happened to the data on its way to the dashboard. To tell which, you need to know that way: every step the number took from its source to the screen.
The rule
Trace a number upstream to fix it, downstream to protect others.
Data lineage is the recorded path of a number: which sources feed it, which steps transform it, and which reports use it. For Weftwise's on-time KPI:
Shop platform (orders) Courier feed (deliveries)
| |
raw_orders raw_deliveries <- nightly loads by Kabir
| |
stg_orders stg_deliveries <- cleaned: types, time zone,
\ / test orders removed
+----- fct_deliveries ---+ <- one row per delivered order,
| with an on_time flag
measure: On-time delivery %
|
+----------------+----------------+
| | |
Ops dashboard Monthly report Courier scorecard
for MeeraLineage works in two directions:
- Upstream, towards the sources, when a number looks wrong: where did the error enter?
- Downstream, towards the reports, before or after a change: who will be affected?
Modern BI tools and transformation tools can draw much of this map automatically from the code. Even a hand-drawn version like the one above saves hours the first time something breaks.
The shortcut
Walk one hop at a time and compare with reality. Start at the report and move upstream. At each hop, check a few real cases against the truth. The cause lies where the number first goes wrong, reading from the source down.
The report
The dashboard shows 70% on-time for 2 April.
The trap is patching the report: editing the dashboard's formula to "correct" the dip. The number would look right today and break again tomorrow, and the monthly report and scorecard would still be wrong.
Drills
Answer fast — trust the rule.
What is data lineage?
Kabir plans to rename the column promised_date. Which direction of lineage should Ishaan check first?
A total is correct in raw_orders and stg_orders but wrong in fct_deliveries. Where is the error most likely?
The 2 April figures are being restated. Whom must Ishaan tell?
Remember
- Trace a number upstream to fix it, downstream to protect others.
- Walk one hop at a time and compare with reality; the cause lies where the number first goes wrong.
- Trap: patching the report instead of fixing the source.