Cloud Compute and Storage for ML

Tunebox could buy a GPU server for the churn project. It would sit idle six days a week, fill up with data, and break on a holiday weekend. ML workloads are bursty — a big job once a week, a small steady service the rest of the time — and that shape is exactly what the cloud is good at renting.

The rule

Keep data in buckets; rent big machines briefly; run serving small and steady.

Each of the big three clouds — AWS, Google Cloud and Azure — offers the same three building blocks under different names:

Object storage holds files of any size in buckets, is cheap per gigabyte, and every ML tool can read it. It is the natural home for raw data, the DVC remote, and trained model files.

Compute splits into two very different workloads:

• Training needs a lot of power for a short time. Start a large machine — with GPUs if the model is a neural network — run the job, and shut it down. You pay only for the hours used.
• Serving needs a little power all the time, with low latency. A few small machines, or containers, behind a load balancer.

Spot capacity is spare cloud capacity sold at a large discount, but the provider can take it back with little warning. It suits training jobs that save checkpoints and can resume, and it is a poor fit for a live API.

One more rule: put storage and compute in the same region. Moving large data between regions is slow and is usually billed.

The shortcut

For each workload ask two questions: "Does it run always, or in bursts?" and "Can it survive being interrupted?"

Drills

Answer fast — trust the rule.

Remember

• Keep data in buckets; rent big machines briefly; run serving small and steady.
• Ask: always or bursts? Can it survive interruption?
• Trap: paying for idle GPUs, or putting data and compute in different regions.