1,081 gaps in the price list, and every quote was right

The quoting system has a screen that tells you whether the price list is ready. On a catalogue that had just been loaded and checked against the spreadsheet it came from, item by item, that screen reported 1,081 missing pricing rules. Meanwhile the same system was quoting real jobs, correctly, every day.

So you stop believing the screen. You have to; a warning that contradicts what you can see with your own eyes is not a warning, it is wallpaper. And that is the cost, because about 33 of those 1,081 were real gaps — combinations a customer could order that the system had no price for — and they were buried under a thousand items experience had taught everyone to ignore. A real gap in a price list surfaces as a quotation with a missing line, or a price of nothing, on the day a customer asks for exactly that combination.

What was actually going on

The first assumption was that the loading had gone wrong. It had not. Roughly 1,048 of the 1,081 were produced by the way the screen counted, and only around 33 were real. The screen was not detecting an incomplete catalogue. It was manufacturing one.

The catalogue defines each feature — a top, an undershelf, a splashback — once, for the whole company, so that many products can offer it. Each feature comes in several variants, and the variants belong to the feature, not to any particular product. That is the right way to build it: a splashback is a splashback, and defining it once and pointing thirty products at it is exactly what keeps the number of pricing rules manageable.

The readiness screen then asked, for each product, how many variants exist across all the features that product offers, and treated that as the number of pricing rules that ought to exist. The flaw is in “all”. A feature might have eleven variants across the catalogue as a whole. A given product might offer two of them. The other nine exist because some other product needs them. The screen counted all eleven as required and reported nine gaps that were not gaps — for that feature, on that product, multiplied across 57 products.

The sharing that makes the catalogue cheap to maintain is exactly what made a naive count wrong. They are the same property seen from two ends.

Why this is worse than a screen being merely wrong

A check that over-reports does not fail loudly. It fails by being ignored. The first time an administrator sees 1,081 missing rules on a catalogue that is demonstrably producing correct quotes, they learn the number is not to be trusted. From then on the screen is decoration, and the 33 real gaps sit in that list, correctly identified, where nobody will ever look. A check with a high false-alarm rate is not a partial check. It is a check its users have switched off while it carries on rendering.

What we changed

The question the screen was meant to answer is: for every combination a customer could actually order of this product, is there a price. The set of combinations a customer can order is not the set of variants that exist; it is the set this product has been configured to offer, and the system already knew both. So the count now runs against what each product actually exposes, and a rule written to cover every variant of a feature is treated as covering them, rather than the screen looking for one rule per variant.

There was a second class of false gap underneath. Twelve of what remained pointed at rules the spreadsheet referred to but never defined. Those are real, in the sense that no rule exists, but they are not the administrator’s problem to fix by clicking around — someone has to write the missing rule. They now go to a separate list labelled as gaps in the source data.

Three categories, three lists. Not one number.

What it did not fix

The 33 real gaps still had to be filled by a person who knows the prices, and the twelve undefined rules still had to be written. What changed is that the list is now short enough to finish and honest enough to trust, which is the only state in which anyone finishes it.

The general shape, for anyone with a warning they have stopped reading

This appears wherever a system reports completeness over things shared for reuse. You have options pooled so that many parents can use them, assignments that say which pool members a given parent actually uses, and a count that joins to the pool instead of the assignments — usually because the pool is one join away and the assignments are two, and the difference does not show on small data.

The tell is that the reported number grows with the size of the catalogue rather than with how incomplete it is. If adding a product elsewhere increases the number of gaps reported against an unrelated product, the count is looking at the pool.

Two checks are worth doing before believing any such number. The first is arithmetic: ask what its maximum could be. A screen saying 1,081 missing out of a possible 2,698 rules, on a catalogue quoting real jobs every day, is not credible on its face and should have been challenged before it was built. The second is reconciliation: the number of rules the system thinks should exist has to be derivable independently from the source data. Here the spreadsheet’s mappings expanded to a known number of expected rules and the live data matched it, so any screen reporting a large shortfall was contradicting a figure already verified, and the screen was the thing to look at.

And if a check reports a large number of problems on a system that is visibly working, suspect the check first. It is the cheaper hypothesis to test, and in this case it was the correct one.

Where this ends up

The screen belongs to Sazinga Quote, where it is what an administrator reads before the first quotation goes out — which is why any number it reports has to be reconcilable against the source the catalogue was loaded from.