Kubernetes for Training and Serving
The churn project now has two images: one that trains the model and one that serves it. On a Kubernetes cluster they need very different care. Training should run, finish and free its machine. Serving should never stop. Kubernetes has a different object for each, and picking the wrong one causes strange failures.
The rule
Work that ends is a Job; work that stays is a Deployment.
Training is a Job. A Job runs a container until it exits successfully, retries it if it fails, and then is done. A CronJob creates a Job on a schedule — here, every Sunday at 02:00:
apiVersion: batch/v1
kind: CronJob
metadata:
name: churn-train
spec:
schedule: "0 2 * * 0"
jobTemplate:
spec:
backoffLimit: 2
template:
spec:
restartPolicy: Never
containers:
- name: train
image: tunebox/churn-train:2026-09
resources:
requests: { cpu: "8", memory: 32Gi }Serving is a Deployment. A Deployment keeps a set number of identical pods running, replaces any that crash, and rolls out new image versions gradually:
apiVersion: apps/v1
kind: Deployment
metadata:
name: churn-api
spec:
replicas: 3
selector:
matchLabels: { app: churn-api }
template:
metadata:
labels: { app: churn-api }
spec:
containers:
- name: api
image: tunebox/churn-api:7
ports: [{ containerPort: 8000 }]
resources:
requests: { cpu: 500m, memory: 1Gi }
readinessProbe:
httpGet: { path: /health, port: 8000 }A Service in front of the Deployment spreads requests across the three pods.
Both declare resources, so the scheduler places them on nodes with room. A neural network would add a GPU under limits, such as nvidia.com/gpu: 1, which also requires nodes with GPUs and their device plugin. The readiness probe matters for models: loading a large model takes time, and traffic should only arrive once /health says the model is loaded.
The shortcut
Ask "When this container exits, is that success or a problem?" Success → Job. Problem → Deployment.
churn-train
Exits after writing the model. Exiting is success, so Job, wrapped in a CronJob for the weekly schedule.
Drills
Answer fast — trust the rule.
Which object should run the weekly retraining?
The training script is deployed as a Deployment by mistake. What happens after it finishes?
Why give the churn API a readiness probe?
A pod needs one GPU. What does its spec add?
Remember
- Work that ends is a Job; work that stays is a Deployment.
- Ask whether exiting is success (Job) or a problem (Deployment).
- Trap: running training as a Deployment, which restarts it forever.