Do we need to train our own model, or is prompting enough?
Start with instructions plus your own data retrieved at run time. Training a model is worth considering only when you need highly consistent output on a narrow judgment at large volume, and you already hold a substantial set of correctly labeled examples. Most problems that feel like model problems are context problems - the system was never shown the information it needed to answer.
Three levers, in order of reversibility
Work through them in order, and stop as soon as the output is good enough.
- Instructions and output structure. Cheapest to change, easiest to reverse, and responsible for most early gains. Defining the exact output shape usually helps more than any clever wording.
- Your own data at run time. Retrieving the relevant job history, price book, service definitions or call context and putting it in front of the model. This is where integration work pays off, and it is the step most often skipped.
- Tuning a model on your examples. Real work, real maintenance, and it locks you to a specific base model for as long as you keep it.
The prerequisite people skip
Tuning requires a body of examples that are correctly and consistently labeled. The word doing the work is consistently. If you hand the same fifty calls to three managers and they disagree on the label, tuning will faithfully encode that disagreement and produce output nobody trusts.
Which means the first step toward tuning is the same as the first step toward good evaluation: get humans to agree on what the right answer is, in writing, with examples. Teams often discover during this exercise that the model was never the constraint - the standard was ambiguous.
What tuning genuinely buys
It is good at making output boringly consistent: the same format every time, house terminology, the right level of terseness, a narrow classification applied the same way across enormous volume. It can also let a smaller model do a job a larger one was doing, which matters when volume is high.
It is not a way to teach a model facts about your business that change. Your price book, your service areas and your current promotions belong in retrieved context, not baked into weights, because they change and the model does not.
The model is the least durable part of your system
Base models are replaced regularly, and each replacement can quietly change behavior your workflow depends on. What survives model turnover is everything around it: the connections to your systems, the retrieval layer, the output contract, the evaluation set and the review process.
Build so that swapping the model is a configuration change rather than a rebuild. That is the practical meaning of model integration, and it is why we treat the surrounding system - not the model - as the asset in operational AI work.
Topics: fine-tuning · prompting · model selection · architecture
Have a version of this question about your own business?
The useful answer usually depends on which systems you run and how they're connected. That's a conversation, not a blog post.