Realtime and Hybrid Ingestion
A rider opens the PedalCity map at 08:03 and walks to a dock that showed four bikes. It is empty: the map was built from last night's batch. Some data loses its value within seconds. For that data you need streaming, and usually you keep batch running alongside it.
The rule
Stream when a decision can't wait; batch when it can; hybrid when both.
Realtime (streaming) ingestion handles each event as it arrives. Dock sensors publish readings to a message broker, a system such as Apache Kafka, RabbitMQ or a cloud queue, which holds the events in order. A stream processor reads them continuously and updates results within seconds.
dock sensor ──► broker topic "dock-readings" ──► stream processor ──► live availability table ──► mapStreaming earns its place when someone acts on the data straight away: the rider choosing a dock, or a van driver sent to an empty station. It costs more. The processor runs all day, and late or duplicate events must be handled without the luxury of a finished window.
A middle ground is micro-batching: tiny batches every 30 seconds to a few minutes. Many "realtime" systems actually work this way, and it is often fresh enough.
Hybrid ingestion runs both paths on the same events:
- A fast path streams readings to the live map. It is quick and may be slightly off.
- A batch path lands the same readings in the lake and recomputes accurate hourly counts each night, after all late and duplicate events have settled.
The fast path serves decisions right now. The batch path is the record of truth for reports and model training. When the two disagree, the batch result wins for history.
The shortcut
For each use, ask "What happens if this data is an hour old?" If someone makes a worse decision → stream it. If nothing changes → batch. If the same data serves both a live decision and a careful report → hybrid.
Live map
An hour-old count sends riders to empty docks, so stream it with seconds of delay.
Drills
Answer fast — trust the rule.
Which PedalCity use most needs streaming?
What does a message broker do in a streaming pipeline?
In a hybrid design, the live map and the nightly report show slightly different counts for 08:00. Which is the record of truth for history?
Van drivers are dispatched every 15 minutes. Which approach is fresh enough and simplest?
Remember
- Stream when a decision can't wait, batch when it can, and go hybrid when both apply.
- Ask what happens if the data is an hour old.
- In hybrid designs the fast path serves now, while the batch path is the record of truth.