Hire MLOps Engineers

MLOps earns its place the second time a model is deployed. The first deployment can be done by hand. The tenth, with a rollback at two in the morning and a question about which data version produced the current predictions, cannot. The discipline is mostly about making the boring path the easy one.

Most of this role is ordinary platform engineering applied to models, and the parts that are genuinely specific to machine learning are fewer than the tooling market suggests: versioning a dataset alongside code, promoting an artefact rather than a branch, and monitoring a quality signal that has no exception attached to it.

Instrument the job, not just the model

Every inference run in one of our services writes a row before anything else happens: which provider and model answered, how many attempts it took, how long it took, tokens in and out, whether it used a shared credential, the final status and the error text if there was one. Nothing is backfilled, so a missing row means the job never ran, which is itself information.

That table is what a weekly report reads — success rate by job type and by provider, average attempts, a histogram of how often a retry was needed, and the most common failure strings. It cost an afternoon and it is the only reason anyone can answer “is it getting worse” with a number rather than an impression.

A model you cannot query the history of is a model you can only have opinions about.

A canary that makes real calls

Mocked tests are the right default: they are fast, deterministic and they cover your own logic. They also cannot catch anything about the provider, and the provider is the part that changes without telling you.

So a scheduled job makes one genuine call against every stored credential and exits non-zero if any of them fails, which turns it into an alert. The list of things it has caught reads like an argument for its own existence: a provider rejecting a parameter on a newer model tier that was accepted on the previous one, a model listed by the catalogue endpoint that fails on the chat endpoint and made a perfectly good credential look rejected, and a service beginning to return billing errors the day free credit ran out.

Promotion, rollback and the artefact you actually deployed

Three habits, none of them exotic.

Prefer an artefact format you can inspect over one that executes. Our vision classifier ships its prototype vectors as a plain array archive rather than a pickle, because a pickle is executable code and loading one from a build pipeline is a supply-chain decision, not a serialisation choice.

Verify the thing you are about to promote, not the process that produced it. We learned that outside machine learning, when a default in a build script resolved to a staging host and produced an entire site marked as not-indexable; the deploy now inspects the staged output on the server and refuses to swap if the check fails. The same principle applies to a model: assert something about the artefact — its shape, its version, a known prediction — before it goes live.

And make anything that writes safe by default. Our automated labelling pass is a dry run unless explicitly told otherwise, and it only ever writes to records a person has not already touched.

The operational traps that are not about models at all

Scheduled work arms when a module is imported rather than when a start function runs. That is fine on a server and dangerous on a laptop, where an engineer connecting to a real database to look at something quietly becomes a second writer alongside the live instance. One environment switch that wraps every schedule fixes it.

Backups are claims until restored. Ours are verified by restoring into a scratch database and counting rows, not by checking that a file exists with a plausible size. The same standard should apply to a model registry: a rollback path nobody has walked is a diagram.

Where an MLOps hire is premature

Before the second model, or before the first one has been retrained once. A platform built for a delivery cadence that does not yet exist is a set of abstractions fitted to imagined requirements, and it will be wrong.

Where there is one model, changing rarely, serving a low-stakes decision — a scheduled job, a versioned artefact in object storage and a dashboard will do, and the money is better spent on the data. And where the organisation has no platform engineering at all: this role sits on top of containers, continuous integration and infrastructure as code, and hiring it into a vacuum produces a person maintaining tooling nobody else can operate.

What we interview for

Ask what happens after a bad model reaches production. A candidate who has lived through it will describe the registry, the promotion gate that should have caught it and how long rollback actually took. One who has not will describe a training pipeline.

Then two more. What in your last system was measured continuously that had no exception attached to it — that is what monitoring a model actually means. And what would you delete from the platform you built, because everyone who has built one has an answer.