Serving Models: Batch and Online
Tunebox wants two things from the churn model. The retention team emails at-risk subscribers every Monday. The app wants to show a "we'll miss you" offer the instant someone opens the cancel page. Same model, two very different ways of serving it.
The rule
Ask when the answer is needed: later for many, now for one.
Batch serving scores many rows on a schedule and stores the results. A job runs on Sunday night, loads models:/churn-model@champion, scores every active subscriber and writes the scores to a table. The Monday email reads that table.
- Simple, cheap, easy to retry.
- Predictions can be up to a week old.
Online serving answers one request at a time, behind an API, in milliseconds:
from fastapi import FastAPI
import mlflow, pandas as pd
app = FastAPI()
model = mlflow.pyfunc.load_model("models:/churn-model@champion")
@app.post("/predict")
def predict(features: dict) -> dict:
risk = model.predict(pd.DataFrame([features]))[0]
return {"churn_risk": float(risk)}- Fresh answers using the latest behaviour.
- Must stay up and fast, so it needs monitoring, several replicas and a latency budget.
Both share one danger: training-serving skew, where features are computed differently at serving time than in training. The cure is to use the same feature code in both places.
Releasing a new version safely usually goes in stages:
- Shadow — the new model scores real requests, but its answers are only logged, never used.
- Canary — it serves a small slice, say 10%, of real traffic while the rest stays on the champion.
- Full rollout — if error rate, latency and business metrics hold, it takes all traffic. If not, roll back by moving the registry alias.
The shortcut
For every use of the model, ask "When does someone need this answer, and for how many users?" Later and for many → batch. Right now and for one → online.
Monday email
Needed once a week, for all subscribers. Batch job on Sunday night.
Drills
Answer fast — trust the rule.
A dashboard shows each region's predicted churn, refreshed every morning. Which serving style fits?
During a shadow deployment, what happens to the new model's predictions?
Training computed skips_per_week over four weeks, but the API computes it over one day. What is this called?
A canary at 10% shows higher error rates than the champion. What should happen next?
Remember
- Ask when the answer is needed: later for many is batch, now for one is online.
- Release in stages: shadow, canary, full rollout, with rollback ready.
- Trap: training-serving skew; share one feature codebase between training and serving.