Pebellslearn by building
0%
RoadmapsMatrixCoursesSandboxSign in
The Data Engineering Lifecycle  ›  Lesson

Data Ingestion

Data Ingestion 12 minIngestion moves data from sources into the platform, either pulled or pushed and either as a full copy or incrementally from a saved watermark.

Data Ingestion

Every night PedalCity must bring yesterday's rides, the station list, payment records and dock readings into its data platform. Copying all 15 million rides every night would be slow and wasteful. Copying only new rows would miss the rides that were updated. Ingestion is where these decisions are made.

The rule

Ingestion moves data in: decide push or pull, full or incremental.

Pull or push: who starts the transfer?

  • Pull: your pipeline asks the source. It queries the app database or calls the weather API.
  • Push: the source sends data as it happens. Dock sensors post each reading to an endpoint, or the payment provider calls a webhook.

Full or incremental: how much do you copy?

  • Full load: copy the whole dataset every time. This is simple and right for small tables. PedalCity's stations table has 120 rows, so copying all of it nightly costs nothing.
  • Incremental load: copy only what changed since the last run. To do that you keep a watermark, which is the point you reached last time.

For rides, the watermark is a timestamp. Each run takes the window between the last watermark and now:

SELECT *
FROM rides
WHERE updated_at >  '2026-09-24 02:00:00'
  AND updated_at <= '2026-09-25 02:00:00';

Two details matter. The filter uses updated_at, not created_at, because rides are updated when docked. And the window has a fixed upper bound, so the next run starts exactly where this one ended, with no gaps and no overlaps.

The shortcut

For each source ask two questions: Who starts the transfer? (push or pull) and Is it small and stable, or large and growing? (full or incremental). For incremental loads, name the column that proves a row changed.

Planning PedalCity's nightly ingestionStep 1 of 4

  1. stations

    Pulled from the app database. 120 rows that rarely change, so a full load.

The trap is saving the watermark before the load succeeds. If the load then fails, the next run skips that window forever.

Drills

Answer fast — trust the rule.

quick drill

The stations table has 120 rows and changes a few times a year. Best way to ingest it?

quick drill

Dock sensors send each reading to an endpoint the moment it is taken. What is this?

quick drill

An incremental ride load filters on created_at instead of updated_at. What does it miss?

quick drill

Why give the incremental window a fixed upper bound instead of "everything after the watermark"?

Remember

  • Decide push or pull, and full or incremental.
  • Incremental loads need a watermark and the column that proves a row changed.
  • Save the watermark only after the load succeeds.

Practice — level up

0/1 solved

Write your answer, then compare it with a rubric and a model answer.

ExplainPlan an incremental load

Describe how you would ingest PedalCity's payments table (about 50,000 new or updated rows a day, with refunds updating existing rows). Cover push or pull, full or incremental, which column you filter on, and when the watermark is saved.