APIs as Sources
PedalCity wants each hour's weather next to its rides, and the payment provider only shares refunds through its web API. An API is a door someone else controls. It hands out data a page at a time, limits how often you can knock, and sometimes does not answer at all. A good extractor expects all three.
The rule
APIs hand data out in pages, under limits — plan for both, and for failure.
Pages. An API rarely returns everything at once. It sends a page of results plus a pointer to the next one:
{
"refunds": [
{"id": "rf_301", "ride_id": 88121, "amount": 45.0},
{"id": "rf_302", "ride_id": 88377, "amount": 30.0}
],
"next_cursor": "eyJhZnRlciI6InJmXzMwMiJ9"
}You request again with that cursor until next_cursor comes back empty. Cursor pagination is safer than page numbers. With page numbers, a record inserted while you are paging shifts every later record by one, so you can skip or repeat items.
Limits. Providers cap requests, for example 100 per minute. Go over and you get HTTP status 429 Too Many Requests. The polite response is to wait and retry, doubling the wait each time. This is called exponential backoff.
Failure. Networks time out and servers return 5xx errors. These are usually temporary, so retry them a few times. A 4xx error such as 401 (bad credentials) or 404 (not found) will not fix itself, so stop and alert someone.
Finally, save each raw response before transforming it. If your parsing has a bug, you can reprocess the saved files instead of asking the API for months of history again.
The shortcut
For every API call ask, in order: Is there another page? Am I over the limit? Is this failure temporary? Keep going, wait, retry, or stop.
Request page one
Ask for refunds created yesterday. Save the raw JSON, then read
next_cursor.
Drills
Answer fast — trust the rule.
A call returns HTTP 429. What should the extractor do?
The response's next_cursor is empty. What does that mean?
Why can page-number pagination skip records?
Which error should NOT be retried automatically?
Remember
- APIs hand out data in pages, under limits: plan for both, and for failure.
- Follow cursors until empty, back off on 429, retry temporary errors, stop on client errors.
- Save raw responses so you can reprocess without calling the API again.