The model should never produce the number, and the cheap model still lost
An owner asked us a reasonable question before agreeing to anything: if the sales team could ask the system questions in plain English, what would it cost to run, every month, forever.
That is the question most AI proposals answer with a shrug and a range. We went and measured it, on their data, and the number surprised us in a way that killed our own working theory.
The rule that came first
Before any of the cost work, one design decision: the model never produces a number. It selects a tool and pulls a name out of the question. Finding the rows, applying the scope and summing the column is SQL, running under the asker’s own permissions.
This was not caution for its own sake. It is the difference between a system that is occasionally unhelpful and one that is occasionally, confidently wrong in a board pack.
What we expected, and what the measurement said
Our theory was that with the arithmetic removed, the model’s remaining job — pick one of twenty-three tools, extract the name — is classification, and classification is easy, so the cheapest model on the shelf would do and the bill would round to nothing.
An eight-question routing suite against the live system, priced per thousand questions:
| Model | Correct | Cost per 1,000 questions |
|---|---|---|
| Nova Micro | 3 of 8 | $0.02 |
| Nova Lite | 6 of 8 | $0.04 |
| Nova Pro | 8 of 8 | $0.45 |
The cheap model is not good enough. Micro mostly failed by not calling a tool at all — it wrote a sentence about the question instead, which is not so much a misclassification as declining to play. Lite got six of eight, which sounds close and is not: two wrong answers in eight is a system nobody trusts twice.
So we bought the expensive one. And the owner’s question was still answered well, because of the right-hand column — the entire spread is under half a dollar per thousand questions. Choosing the cheapest model over the best one saves forty-three cents per thousand. That is not a decision; it is a rounding error with a quality penalty attached.
The useful finding was not “use the cheap model”. It was that the model bill is not where this project’s money goes, so stop optimising it and buy accuracy. The cost sits in the schema work, the scoping and the testing, all of which is engineering time, and none of which a model choice changes.
An earlier version of this article argued the opposite — that the cheap model would be sufficient, and that the measurement had not yet been made. It had been made, three days before I wrote it, and it refuted the theory. The conclusion the owner needed survives; my reasoning for it did not. Leaving that visible seems more useful than quietly correcting it.
Where the wrong number would have come from anyway
Buying the strongest model does not remove the problem the design rule exists for. Running it against real data caught it inventing figures.
Asked to total ten rows, it returned 3,585 on one run and 3,485 on the next, when the true figure is 3,374. Three answers, one correct, no error anywhere. That is the failure mode that ends these projects, and it came from the best model on the list.
The fix was structural rather than a sterner instruction. The server computes column totals itself, over the exact rows it returned, and the prompt forbids the model from doing arithmetic at all. You cannot get a wrong sum out of something that is not permitted to add.
What is still wrong with it
One class of invention survives and is not fixed. Superlatives at the extremes are now computed on the server, but ordinals in between are still made up — in one run it reported a product as second-highest by volume when it was sixth, because it read position in the table it had been handed as rank by the measure it was naming.
There is also a routing defect: asking about a named person in the most natural phrasing sends the question to the wrong resolver, finds nothing, and produces a refusal whose text is false. Adding one word makes it work, so it is routing rather than data, and it is open.
Both are here because a buyer’s real question is not whether a system has defects. It is whether the people who built it can tell you what they are.
Where it actually is
Running end to end on the client’s QA environment: twenty-three tools, nineteen scoped SQL functions, the permission gate, the portal screen, and an automated browser test that signs in as real users and checks that two people asking the same question get different rows. It is not in production, and it will not be until the two defects above are closed.
The cost answer we gave the owner: the model is the cheapest part of this, and anyone quoting a large monthly figure for it is quoting for something else.
If you are costing one of these for your own business, the shape of the work and what it is measured against is set out under AI data assistants.