67,000 photos already labelled by humans, no GPU, and no model training

Field staff photograph vehicles at handover and return. Each photo is supposed to be tagged with what it shows — odometer, fuel gauge, a particular exterior view, the signed checklist, damage. Many arrive untagged, and somebody has to go through them.

The obvious solution is an image classifier. The obvious objection is that training one requires a labelled dataset, a GPU, and a couple of weeks.

Two facts made a much cheaper approach available. First, there were around 67,000 photos already tagged by humans, as a byproduct of years of ordinary operations. Second, the only available server had eight cores, no GPU, and was already running the live application and two databases — training anything on it would have pegged every core for hours.

The approach

Embed all 67,000 existing photos once with a general-purpose vision-language model. Average the embeddings within each tag to produce one prototype vector per photo type — 24 of them. To classify a new photo, embed it and find the nearest prototype by cosine similarity.

That is the entire method. No training, no fine-tuning, no labelled/unlabelled split to manage, no hyperparameters. The embedding pass is the only expensive step and it runs once. Adding a new category later means averaging the embeddings of its examples, which takes seconds.

It reuses the labels without any training, which is the point. Most organisations sitting on a classification problem are also sitting on years of human decisions that were recorded for operational reasons and never thought of as training data. Support tickets with categories. Documents with types. Transactions with codes. Before commissioning a model, it is worth asking what has already been labelled by the ordinary course of business.

The infrastructure decisions followed the same logic of taking the cheap path where it costs nothing: the classification runs off-server, pulling only two lightweight read-only queries over a secure connection and fetching the images directly from object storage, so the production server is never in the image path. It has a load-average guard, rate-limited writes, and it only ever touches photos that are still untagged. It cannot overwrite a human’s tag.

That last constraint is worth stating as a rule of its own. A system that suggests should never be able to overwrite a decision a person already made. It is a one-line condition and it removes an entire category of trust problem.

Measuring it honestly

Accuracy was measured on a held-out set of already-tagged production photos that were not used to build the prototypes.

Applying the classifier to everything gave 70.6% accuracy. Which, on its own, is not good enough to be useful — three wrong tags in ten would create more work than it saves.

But accuracy across everything is the wrong number, because the classifier also produces a margin: the gap in similarity between the best-matching prototype and the second-best. A large margin means the photo sits clearly nearer one prototype. A small margin means two prototypes are nearly equally close, and the model is effectively guessing between them.

Filtering by margin:

• margin above 0.015: 86.9% accurate, covering 55% of photos
• margin above 0.025: 89.6% accurate
• margin above 0.04: 96.1% accurate, covering 30% of photos

So the system can be tuned to whatever accuracy the operation needs, by accepting less coverage. At the tightest threshold it correctly tags three photos in ten and leaves the rest to a person, at an accuracy where nobody has to check its work.

This is the single most useful design move in applied machine learning, and it is routinely skipped. A model that outputs a class is a model you must either trust or verify. A model that outputs a class and a confidence, with a threshold that routes low confidence to a human, is a tool that reduces work by a known amount at a known error rate. The second is dramatically more valuable than the first, and it usually requires no additional modelling — the confidence signal is already there in the scores, and people throw it away by taking the argmax.

Knowing the failure mode is part of the design

Per-category accuracy told a more interesting story than the aggregate.

Odometer and fuel readings, the signature capture and the checklist were near 100%. These are visually distinctive and consistent — every one looks broadly like every other one.

Damage close-ups were 36%. Damage is not a visual category in any coherent sense; a scratched bumper and a cracked windscreen have nothing in common except an intention. One prototype vector cannot represent that, and the honest conclusion recorded was to leave that category to a human rather than attempting to improve it.

The most instructive finding was about the exterior views. The model reliably identified which view a photo showed but frequently could not tell left from right — mirror symmetry is a well-documented weakness of this class of model.

And here is where the design absorbs the flaw rather than fighting it. Because the left and right prototypes sit close together in the embedding space, a photo that is ambiguous between them produces a small margin. The threshold that already exists sends it to a person. The known weakness routes itself to the right place with no special-case code, because the confidence measure happens to encode exactly that uncertainty.

That is worth generalising. When you understand a model’s failure mode, check whether your confidence measure already separates it. Often it does, and the fix is a threshold rather than a feature.

The project’s documentation ships the measured accuracy table rather than a claim about performance. That is a small discipline with a large effect on whether anybody downstream can make a sensible decision about where to use it.

What this says about AI in delivery

The genuinely useful question is not “can a model do this” but “what is the cheapest thing that turns work people have already done into work they no longer have to do”.

Here the answer had four parts, and none of them was a model architecture:

• Existing labels, produced as a byproduct of work already done.
• A general-purpose embedding model used off the shelf, with no training.
• A confidence measure used to decide what to attempt rather than what to answer.
• A hard constraint that it can never overwrite a person.

The result is not state of the art at anything. It correctly handles a portion of a tedious task at an accuracy where nobody needs to check it, and it declines the rest.

That is usually what “AI in production” should look like in a small operation: not autonomy, but a measured reduction in volume, with the residue routed to the people who were doing all of it before. The thing to negotiate up front is not accuracy — it is what happens to the cases the system is not confident about, and who decides where that line sits.