Handling Incidents
It's 8:20 pm on the first night of the Diwali sale when the alert fires: card payments are failing for about a third of customers. Within five minutes eight people have joined the call, three of them are reading the same logs, and two directors are asking for updates in different channels. Nobody is in charge.
The rule
Restore service first, find the cause later; one person leads.
An incident is something that is hurting customers right now. Most teams grade incidents by severity: for example, SEV1 for "checkout is broken for many customers" and SEV3 for "a minor feature is misbehaving". Severity decides how many people get pulled in and how often updates go out.
Clear roles make an incident calmer and shorter:
- Incident lead. Coordinates, assigns work and makes calls such as "roll back now". The lead deliberately doesn't debug, because someone has to hold the whole picture.
- Communicator. Posts updates to support, leadership and other teams on a fixed schedule, so the responders aren't interrupted.
- Responders. Investigate and fix, reporting what they find to the lead.
Mitigate before you diagnose. Rolling back a deploy, switching off a feature flag or failing over to a backup can stop the damage before anyone knows the exact cause. Finding the cause matters, but it happens after customers can pay again.
The engineering manager often takes the communicator role, or makes sure someone does. Meera also shields the responders from "any update?" messages and watches for exhaustion on long incidents, handing roles to fresh people when needed. Once service is back, the learning happens in a postmortem, which is the next lesson.
The shortcut
In the first minutes, ask two questions: "What changed?" and "Can we undo it?" Most incidents follow a change, such as a deploy, a config edit or a traffic spike, and undoing that change is usually the fastest way to stop the damage.
Declare and assign
Arjun, who is on call, declares a SEV1 at 8:25. Farhan takes incident lead, Meera takes communications, and Priya and Arjun investigate. Everyone else leaves the call.
Drills
Answer fast — trust the rule.
Checkout errors spike right after a deploy. What is the first move?
What does the incident lead mainly do?
Five stakeholders keep messaging the responders for updates. What is the best fix?
The incident is in its fourth hour and it is past midnight. What should Meera watch most closely?
Remember
- Restore service first, find the cause later; one person leads.
- Ask "what changed?" and "can we undo it?" in the first minutes.
- Separate the roles: a lead who doesn't debug, a communicator on a schedule, and responders who fix.