MLOps: what happens the day after the model works

MLOps is the set of practices that keep a machine learning model working six months after someone demoed it in a slide deck. It is not a tool you buy: it is the boring part — versioning, deployment, monitoring, retraining and accountability — that separates a nice experiment from a system the business depends on.

The scene repeats itself. A vendor delivers a forecasting model with good accuracy on historical data. Management signs it off. Three months later nobody can say whether it is still right, who checks it, what data it was trained on, or what happens if tomorrow it returns nonsense. The model did not break suddenly. It was left alone.

This guide is for IT leads, management and data teams who already have a model — or are about to commission one — and want to know what has to be built around it so it does not turn into technical debt. If you are earlier than that, start with machine learning for business.

What is MLOps and how is it different from DevOps?

DevOps solves how to ship code to production repeatably: you control the code, test it, deploy it, observe it. MLOps adds two things classic software does not have:

• Behaviour depends on data, not only on code. The same code trained on January data and on October data gives different results. Versioning the code is not enough: you have to version the data and the trained model too.
• There is no "right" or "wrong", there is "how wrong". An endpoint returns 200 OK and the prediction can still be systematically off. Traditional availability alerts never catch it.

Hence the practical definition: MLOps is DevOps plus data traceability plus surveillance of prediction quality. Everything else is tooling.

One detail that saves arguments: MLOps does not require an expensive platform. A model that runs once a day over 40,000 rows can be operated with a git repository, a container, a scheduler and a results table. What you cannot do is operate it without knowing which version is running.

What changes when the model leaves the demo?

On the data scientist's laptop the data arrives clean, in one file, all at once, with no deadline. In production all four of those conditions change.

| In the demo | In production | |---|---| | A hand-prepared CSV | Data from the ERP, with new fields and unexpected nulls | | Training and prediction in one place | Monthly training, hourly prediction, different code on each side | | Error measured once | Error has to be measured weekly against what actually happened | | If it fails, you rerun it | If it fails, someone in operations sees a gap on screen and has no one to call | | Nobody asks why | A client or an auditor asks why that decision was made |

The most expensive difference is the third row. Measuring error in production forces you to store every prediction with its date and, when the real outcome arrives, to match them up. If that was not designed in from the start, reconstructing it later is weeks of work.

If you cannot answer "how wrong was the model last month?" with a single query, you do not have a model in production. You have a script that runs.

The second silent change is training-serving skew: the data transformation gets written twice, in two places, for training and for serving, and the two drift apart. It is the number one cause of models that perform worse in the real world than in testing. There is only one way to avoid it: the same preparation code serves both training and prediction.

Which parts of MLOps do you actually need?

You do not need the full platform. You need these five questions to have a written answer.

| Part | The question it answers | Minimum viable | |---|---|---| | Model and data versioning | Which version is running and what was it trained on? | A register with: version id, date, training data range, metrics | | Reproducibility | Can I retrain this identically a year from now? | Code in git + pinned dependencies + dataset snapshot | | Deploy and rollback | How do I publish a new version and revert it in 10 minutes? | Two versions coexisting and a switch | | Monitoring | Is the prediction degrading? | Weekly error metric + drift alert on the inputs | | Governance | Who approves a retrain and who answers for a failure? | A name against each item, written down |

If you have to start with one, start with monitoring: it turns an invisible problem into a problem with a date on it. The rest can be built on top.

None of this works if the input data is garbage. MLOps does not fix an ERP with duplicated customer records; it exposes the problem faster. Before investing here, look at data quality.

How do you spot a model that is degrading?

A model never warns you. It keeps returning numbers with complete confidence while the world it learned stops existing. There are three signals and all three are worth watching, because they arrive at different moments.

1. Input drift (arrives first). The distribution of what comes in changes: a new product that did not exist during training, 30% of rows carrying an unknown sales channel, average order value up 40% because prices went up. You detect it by comparing each variable's distribution over the last week against the training set. It does not prove the model is failing, but it is the earliest warning you will get.

2. Prediction drift (arrives next). The spread of what the model predicts shifts. A lead scoring model that used to flag 8% of leads as hot starts flagging 22%. It may be real or the model may be lost. It is cheap to watch because you do not need the actual outcome.

3. Performance drop (arrives late, and it is the one that truly matters). Real error goes up. You can only measure it once the outcome lands: for demand forecasting, at month close; for a churn model, three or six months later. That is exactly why the first two signals are not optional: they are all you have while you wait.

A sensible setup for a mid-sized company: an automated weekly report with input and prediction distributions, and a monthly report with real error against what happened. Two recipients: whoever maintains the model and whoever uses the output. If only the technical person reads it, degradation gets discussed too late.

Set the threshold before you need it. "If average error exceeds 18% two months running, we retrain and tell operations." Written down. Without a written threshold, every month is decided on gut feel, and gut feel always says it will hold one more month.

How often should you retrain a model?

The honest answer: it depends how fast your business changes, and you find out by measuring, not by deciding in a meeting. Three patterns cover almost every case:

• On a calendar. Retrain monthly or quarterly, no matter what. Simplest option and enough for stable demand, maintenance or segmentation. Upside: predictable and plannable. Downside: you retrain when it is unnecessary and you are late when it is not.
• On a trigger. Retrain when monitoring crosses the threshold. More efficient, but it requires reliable monitoring and someone who responds to the alert.
• On a business event. Catalogue change, merger, new product line, price change, major campaign. No metric catches these in time: the business knows before the system does. You have to ask them to tell you.

And an unpopular warning: retraining does not always improve things. If the new data is contaminated — because the model itself influenced it, because it was an atypical month, because the way a field is recorded changed — the retrained model can be worse. So every retrain goes through the same gate: evaluate against a stable validation set, compare against the version in production, and only deploy if it wins. If it does not win, the old one stays and you investigate.

The pattern that works best is boring: new model trained, run in shadow mode for a week or two (it predicts, nobody acts on it), compared against the one in production, then swapped in with immediate rollback available.

Who answers when the model fails?

This question sinks more projects than any technical detail. A model in production carries at least four distinct responsibilities, and they are almost never assigned:

• The input data. Who answers for the ERP continuing to deliver the agreed fields. Usually IT or the process owner.
• Execution. Who answers for the job running, and for somebody finding out when it does not. Operations or IT.
• Prediction quality. Who reads the monthly report, decides whether to retrain and takes the heat if the number is wrong. The most frequently forgotten role.
• The decision. Who uses the output and is free to ignore it. Sales, procurement, production. The model recommends; the person decides and owns the decision.

Write it as a four-row table with one name per row. Not a department: a name. And add a degradation procedure: if the model is unavailable or its numbers are not credible, what happens meanwhile? The answer is almost always "we go back to the manual rule we used before", and that needs to be written down and tested, not improvised at eight on a Monday morning.

If the vendor handing you the model does not also hand you those four rows, they handed you half a project. It is the same logic behind choosing an AI consultancy: whoever builds it should answer for what happens afterwards.

What does it cost to keep a model alive?

Nobody budgets for this, and it is the crack the overspend comes through. A reasonable field rule: annual maintenance of a model in production runs between 15% and 30% of what it cost to build, spread across three places:

• Infrastructure. Usually the cheapest part. A tabular model running once a day costs tens of euros a month. Costs spike with real-time inference and with large models.
• People. The real line item. Hours for the monthly review, retrains, investigating an alert, updating dependencies. Between half a day and two days a month per model, depending on criticality.
• Environment changes. The ERP renames a field, a data provider changes a format, a library version has to be migrated. Unpredictable but recurring work: budget for it even though you cannot say what it will be spent on.

From which comes the uncomfortable, useful question: if the model delivers less value per year than it costs to maintain, the right move is to switch it off. A model switched off is not a failure; three zombie models nobody looks at that still feed reports definitely are. To put numbers on that, measuring AI ROI comes first.

Frequently asked questions

Is MLOps the same as having a data scientist?

No. A data scientist builds the model; MLOps is what lets you operate it when that person is on holiday or no longer works with you. Many projects fail precisely because all the knowledge about the model lives in one head and one unversioned notebook.

Do I need an MLOps platform to get started?

Not for your first or second model. Git, containers, a task scheduler and a table to store predictions and metrics are enough to operate a model with proper guarantees. Platforms pay off once you have several models, several teams and need centralised governance.

How often should a production model be reviewed?

At minimum a monthly metrics review and a deeper quarterly review that asks whether the use case still makes sense. For models touching daily decisions or money, automated surveillance should be weekly.

Does the same apply to language models?

The framework applies and the metrics change. There is no clean numeric error, so you evaluate against a fixed set of test cases plus sampled human review. What does not change is the core: version what is in production, measure on a schedule, and have someone accountable.

Where to start

If you already have a model running, the first step is not building a platform: it is writing answers to five questions — which version runs, what data trained it, how its error is measured, when it gets retrained, and who answers for it. The blanks are exactly your work plan.

At adask we build this layer on top of what the company already has, without swapping the stack for the sake of it, and we leave in writing who does what. If you want to know the state of what you already have in production, ask for the audit; if you would rather talk it through, let's talk.