Distributed Systems
DabbaGo splits its app across several machines: an order service, a payment service, a rider service. One lunch time, the order service asks the payment service to charge ₹180, waits three seconds, hears nothing, and tries again. The customer is charged twice. On one machine, calling a function either works or throws an error. Across a network, there is a third outcome, "no idea", and it changes everything.
The rule
The network is unreliable; design every call for silence.
A distributed system is any system whose parts run on different machines and cooperate by sending messages. Almost every modern system is one: a phone and a server, a server and its database, services talking to services.
What makes them hard is partial failure. One machine can be down, slow or cut off while the others keep running. Engineers new to this often assume things that are not true, such as:
- the network always delivers messages,
- messages arrive instantly,
- bandwidth is unlimited,
- the network is secure,
- the set of machines never changes.
Each false assumption leads to a real bug. The worst is treating a timeout as a failure. A timeout only means no reply arrived in time. The request may never have arrived, may have succeeded with the reply lost, or may still be running.
Standard tools for designing around silence:
- Timeouts on every call, so one slow service cannot freeze everyone waiting on it.
- Retries with backoff: try again after growing gaps (1 s, 2 s, 4 s), and only a few times.
- Idempotency: design operations so doing them twice has the same effect as once. The caller sends a unique key; the receiver remembers keys it has already handled.
- Graceful degradation: when a dependency is down, do something useful anyway, such as showing yesterday's popular dishes if recommendations fail.
The shortcut
For every network call, run the silence test: "If this call gets no reply, what do I do, and is it safe to do it twice?"
Ask what silence means
After 3 seconds with no reply, the customer might or might not have been charged. We cannot tell.
The trap is adding retries without idempotency. Retries make silence survivable only when repeating an operation is harmless; otherwise they multiply the damage.
Drills
Answer fast — trust the rule.
The order service's call to the payment service times out. What does it know for certain?
Which property makes it safe to retry the payment call?
Why retry with growing gaps (1 s, 2 s, 4 s) instead of immediately and endlessly?
The recommendations service is down at lunch. What does graceful degradation look like?
Remember
- The network is unreliable; design every call for silence.
- Shortcut: the silence test: "no reply, then what, and is it safe to do twice?"
- Trap: retries without idempotency, which turn one charge into two.