Keeping It Running
Sprout is live. At 3 a.m. the Node process crashes on an input nobody expected, or it keeps running but loses its database connection and fails every request. Nobody is awake. Production needs layers that notice problems and recover on their own, and that wake a person only when they cannot.
The rule
Restart what crashes, test what runs, alert a human when both fail.
Layer 1: restart crashes. On Linux, the service manager systemd starts Sprout's API and watches it. These are the key lines of its unit file, sprout-api.service:
[Service]
ExecStart=/usr/bin/node /srv/sprout/server/src/index.js
EnvironmentFile=/etc/sprout/api.env
User=sprout
Restart=alwaysRestart=always starts the process again whenever it exits. User=sprout runs it as an ordinary user, not root. Everything the app prints goes to the system journal, where journalctl -u sprout-api shows it.
Layer 2: test what runs. A running process is not the same as a working one. A health route checks the things the app really depends on:
app.get('/api/health', async (req, res) => {
try {
await pool.query('SELECT 1');
res.json({ ok: true });
} catch {
res.status(503).json({ ok: false });
}
});Layer 3: watch and alert. Monit is a small Linux tool that checks services on a schedule, restarts them when a check fails, and sends an alert when restarting does not help:
check process sprout-api matching "src/index.js"
start program = "/usr/bin/systemctl start sprout-api"
stop program = "/usr/bin/systemctl stop sprout-api"
if failed port 3000 protocol http request "/api/health" then restart
if 3 restarts within 5 cycles then alertEach cycle (every 30 seconds, say), Monit calls the health route. A failure earns a restart. Three restarts in five cycles means something deeper is wrong, such as the database being down, and Monit emails you. Monit can also watch disk space, memory and CPU, which catches the slow failures, like logs filling a disk, before they become outages.
The shortcut
List the ways the app can fail and ask of each: "What would notice this?" A crash: systemd. A hang or a lost database connection: the health check, via Monit. Something restarts cannot fix: an alert to a person. A failure with no watcher is an outage nobody hears about until users complain.
03:00:00
The database connection drops. Node is still running, so systemd sees nothing wrong, but every request now fails.
Drills
Answer fast — trust the rule.
The Node process exits with an error. What brings it back first?
Why does /api/health run SELECT 1 instead of simply answering "ok"?
Where do you read the Sprout API's output when systemd runs it?
Monit's restarts keep failing. What should happen next?
Remember
- Restart what crashes, test what runs, alert a human when both fail.
- Shortcut: for every way it can fail, name what would notice.
- Trap: a health route that always says "ok". Test the real dependencies.