Batch Ingestion
Most of PedalCity's pipelines run once a night. At 02:00 a job collects every ride from the previous day and processes them together. That is batch processing: it is the oldest, simplest and still the most common way data moves. Its one real cost is waiting.
The rule
Batch gathers data over a window, then processes it all at once.
A batch job is defined by three things:
- Window: which slice of data one run covers, such as "all rides that ended on 24 September".
- Schedule: when the run starts, such as daily at 02:00. It starts after the window closes, with a margin for late rows.
- Latency: how old the newest data can be when it becomes available. Worst case, this is roughly the window, plus the wait before the run, plus the run time.
PedalCity's nightly job covers midnight to midnight, starts at 02:00 and takes 30 minutes. A ride ending at 23:50 on Sunday is available at about 02:30 on Monday, less than three hours later. A ride ending at 00:05 on Monday has to wait for Tuesday's run, so it is available at about 02:30 on Tuesday, more than 26 hours later.
Batch jobs usually write one partition per window, which is a slice of a table or a folder for that date:
INSERT INTO fact_rides
SELECT ...
FROM stg_rides
WHERE ended_at >= '2026-09-24' AND ended_at < '2026-09-25';Why batch is so popular:
- Simple: a job starts, runs and stops. There is nothing to keep alive in between.
- Cheap: compute runs for 30 minutes, not all day.
- Easy to rerun: if Tuesday's logic had a bug, fix it and rerun Tuesday's window.
Its weakness is staleness. Anything that needs minutes-fresh data, such as a live map of free bikes, cannot wait for 02:00.
The shortcut
Ask "What is the oldest data anyone can tolerate?" If the answer is hours or days, batch is the default. Then size the window to match: daily windows for daily reports, hourly windows if people need data within a couple of hours.
Tolerance
Ops reads yesterday's numbers at 08:00, so data up to six hours old is fine.
Drills
Answer fast — trust the rule.
A daily batch covers midnight to midnight, runs at 02:00 and takes 30 minutes. When is a ride that ended at 23:50 on Sunday available?
With the same job, when is a ride that ended at 00:05 on Monday available?
Which need is a poor fit for a nightly batch?
Why is batch easy to rerun after fixing a bug?
Remember
- Batch gathers data over a window, then processes it all at once.
- Ask what the oldest tolerable data is. Hours or days means batch.
- Batch is simple, cheap and easy to rerun, but it is stale by design.