Metrics with Prometheus and Grafana
You now know what to watch on the churn API. The last step is the plumbing: getting numbers out of the running service, storing them over time, drawing them, and waking someone up when one crosses a line. Prometheus and Grafana are the most common open-source pair for this.
The rule
The service exposes, Prometheus scrapes, Grafana shows and alerts.
The service exposes. The Python client library adds counters and timers to your code and serves them as plain text on /metrics:
from prometheus_client import Counter, Histogram, start_http_server
PREDICTIONS = Counter("churn_predictions_total", "Predictions served", ["outcome"])
LATENCY = Histogram("churn_prediction_seconds", "Time to predict one user")
start_http_server(9100) # /metrics on port 9100
@LATENCY.time()
def predict(features):
risk = model.predict(features)[0]
PREDICTIONS.labels(outcome="churn" if risk > 0.5 else "stay").inc()
return riskThree metric types cover most needs:
- Counter — only goes up: predictions served, errors. You look at its rate.
- Gauge — goes up and down: memory used, today's missing-value share.
- Histogram — counts observations into buckets, so you can estimate percentiles such as p95 latency.
Prometheus scrapes. Prometheus pulls: every 15 seconds, say, it requests /metrics from each target and stores every value as a time series — a metric name plus labels, with a value at each timestamp. You query it with PromQL:
histogram_quantile(0.95, rate(churn_prediction_seconds_bucket[5m]))That line reads: "the 95th-percentile prediction time over the last five minutes".
Grafana shows and alerts. Grafana connects to Prometheus as a data source, runs queries like the one above, and draws them on dashboards. Alert rules fire when a query crosses a threshold for a set time, and send a notification to chat, email or an on-call tool.
The shortcut
For any signal, fill in one line: "Record it as a ___, query it as ___, alert when ___ for ___."
Record
A Counter
churn_predictions_totalwith labeloutcome, incremented on every prediction.
The trap is alerting on a single scrape. Short spikes are normal; the "for 30 minutes" part keeps alerts meaningful.
Drills
Answer fast — trust the rule.
How does Prometheus usually collect metrics from the churn API?
Which metric type fits "number of predictions served"?
You need p95 prediction latency. Which metric type should record the timings?
Why add a duration such as "for 10 minutes" to an alert rule?
Remember
- The service exposes, Prometheus scrapes, Grafana shows and alerts.
- For each signal: record as a type, query it, alert when it crosses a line for a while.
- Trap: alerts on single samples, which bury real problems under noise.