Algorithms and Design for Real Data
The first version of the matching step compared every invoice with every shipment. On a 50-invoice sample it finished instantly. On a real month, 6,000 invoices against 200,000 shipment records, it crawled, and it would only get slower as the data grew. Real customer data is bigger and messier than any demo, and it punishes careless algorithms and fragile designs.
The rule
Index once, look up fast; assume every step can fail and repeat.
The algorithm. Comparing each of 6,000 invoices with each of 200,000 shipments is 6,000 × 200,000 = 1.2 billion comparisons. Instead, build a dictionary keyed on carrier and LR number once, then look each invoice up:
by_key = {(s.carrier, s.lr_number): s for s in shipments} # built once
for inv in invoices: # 6,000 look-ups
shipment = by_key.get((inv.carrier, inv.lr_number)) # fast on average
if shipment is None:
flag(inv, "no matching shipment")That is about 206,000 steps instead of 1.2 billion. The work now grows in step with the data instead of multiplying.
The design. The pipeline has stages: fetch mail → read the invoice → match → check rates → review queue. In real life the mail server times out, the invoice-reading service limits how many requests you may send, and carriers resend the same invoice. So:
- Queues between stages, so a slow stage makes work wait instead of losing it.
- Retries with growing waits for temporary failures.
- Idempotency. Each invoice is recorded under a key, carrier plus invoice number, and a key already seen is not processed again. Handling it twice has the same effect as handling it once.
- A dead-letter queue. After a few failed attempts, the item is parked with its error for a person to look at, instead of being retried forever.
The shortcut
Two habits: multiply before you code (sizes times operations gives you the cost) and ask "what if this runs twice?" of every step.
Size it
6,000 invoices a month, checked against 200,000 shipment records.
The trap is trusting the sample. Code that works on 50 rows and a pipeline that only works when nothing goes wrong have both skipped the two habits.
Drills
Answer fast — trust the rule.
Nested loops compare 6,000 invoices with 200,000 shipments. About how many comparisons is that?
Once a dictionary is keyed on carrier and LR number, how does the cost of looking up one invoice change as the number of shipments grows?
A carrier emails the same invoice twice. Which design property stops a second dispute being raised?
The invoice-reading service sometimes times out. What is the best handling?
Why put a queue between the reading step and the matching step?
Remember
- Index once, look up fast; assume every step can fail and repeat.
- Shortcut: multiply before you code, and ask "what if this runs twice?"
- Trap: code that only works on a small sample, and pipelines that only work when nothing goes wrong.