Logs, Mobile Apps and IoT
PedalCity's dock sensors report every 30 seconds, the mobile app records taps, and the servers write log lines. This is event data: a record of something that happened at a moment in time. It is the largest source PedalCity has, and the least tidy. A phone in a tunnel keeps its events for an hour. A sensor that did not hear an acknowledgement sends the same reading twice.
The rule
Events arrive late, twice, and out of order — design for it.
Every event carries two times:
- Event time: when it actually happened, stamped by the device.
- Arrival time: when your pipeline received it.
They can be far apart. A rider taps "unlock" at 08:02 in an underground car park, and the phone uploads the event at 09:10. Count by arrival time and the tap lands in the 09:00 hour, so 08:00 looks too quiet and 09:00 too busy.
{"event_id": "e-7f3a", "type": "unlock_tapped", "rider_id": 5521,
"event_time": "2026-09-24T08:02:11+05:30",
"arrival_time": "2026-09-24T09:10:57+05:30"}Three habits handle the mess:
- Deduplicate by a unique ID. Give each event an
event_idat the source. If the same ID arrives twice, keep one copy. - Group by event time, not arrival time, when the question is about when things happened.
- Allow lateness. Keep each hour's result open for a while, say two hours, and recompute it when late events arrive.
Logs add one more step. Server logs are text lines that must be parsed into fields before they are useful, and lines can be malformed:
2026-09-24 08:14:03 INFO dock=17-04 bike=903 action=dockedDevice clocks also drift. A sensor with a wrong clock can send "future" events, so check timestamps against sensible bounds.
The shortcut
For every event stream ask: What is the unique ID? Which time do I group by? How late can it be?
Unique ID
Each tap carries
event_id. Two copies ofe-7f3acount as one tap.
Drills
Answer fast — trust the rule.
A sensor resends a reading because it missed the acknowledgement. How do you avoid counting it twice?
"How many unlock taps happened between 08:00 and 09:00?" Which time should you group by?
A tap at 08:02 arrives at 09:10. The 08:00 hourly count was first published at 09:05. What should the pipeline do?
Why must server log lines be parsed before analysis?
Remember
- Events arrive late, twice, and out of order: design for it.
- Ask: unique ID, which time to group by, how late it can be.
- Event time says when it happened; arrival time only says when you heard.