Training, Fine-tuning and Inference

Pedal Point's owner asks a fair question: "If the model doesn't know our policies, can't we just teach it?" You can, but that word "teach" hides two very different activities with very different costs. Separating them is the first design decision in almost every AI project.

The rule

Training changes the weights; inference only uses them.

A model is, at heart, a huge set of numbers called weights. Everything it can do is stored in them.

• Pretraining is where the weights come from. The model reads a vast amount of text and adjusts its weights, over and over, to predict the next token better. It takes enormous compute and happens once per model, usually at a big lab.
• Fine-tuning is a smaller round of training on your own examples. It nudges existing weights so the model picks up a behaviour: a tone, a format, a narrow task. It still needs a dataset, a training run and a new model to deploy.
• Inference is using the model. Each customer message is one inference: the weights stay frozen, the model reads the input and produces output.

As an AI engineer you pay for inference on every request, usually per token read and written, and your users wait for it. That is why inference cost and latency (time to respond) matter so much in product design.

A rule of thumb: fine-tune for behaviour, supply facts at inference. Facts change; weights are slow and costly to change. A returns policy is a fact. "Always answer in two short sentences" is a behaviour.

The shortcut

For any proposed change, ask: "Does this step update the weights?" If yes, it is training and needs data, compute and a redeploy. If no, it is inference-time work, and you can change it today.

The trap is reaching for fine-tuning first. Most early problems are fixed faster by better instructions and better data in the request.

Drills

Answer fast — trust the rule.

Remember

• Training changes the weights; inference only uses them.
• Fine-tune for behaviour; supply facts at inference.
• Inference is where you pay and where users wait.