Compliance dashboard showed 100% where it was 1%
A compliance dashboard guessed the unit of a stored figure. A true 1 per cent showed as 100 per cent, so the site that needed the alarm looked perfect.
The headline figure on your compliance dashboard is a percentage. For one organisation, where almost nobody holds the certificates they are required to hold, it showed 100%.
That looks like good news, and good news does not generate tickets. Nobody questioned it. In a compliance product a claim of a hundred per cent is the most expensive claim you can make, because it is the one you would show an auditor, or rely on when deciding who is cleared to work. The true figure was one per cent.
What was actually going on
Between the stored number and the screen sat a short piece of logic that decided what unit the stored number was in. In plain words: if the number is between zero and one, treat it as a fraction and multiply by a hundred. Otherwise treat it as a percentage and leave it.
So a stored value of one, meaning one per cent, appeared as 100%. A stored value of half appeared as 50%. The failure runs in the wrong direction: the worse the real figure, the more likely it is read as a fraction, and the better it looks. Zero came out right only by accident.
The logic existed because the storage did not settle the question. The column allowed any value up to 999.999, and three decimal places suit a percentage with fine detail as well as a fraction to a thousandth. Different writers over the life of the project could plausibly have written either: there was a seeding script, a backfill script and an intended aggregation job. Somebody found a value that looked absurd on screen, wrote a defensive correction so the dashboard would not show it, and the dashboard stopped showing it. The screen looked right afterwards, so nobody asked which writer had been wrong. The tolerance became the specification.
There was a second fault inside the first. The fraction path rounded to two decimal places and the percentage path to three, so the same underlying value came out with different precision depending on which way it arrived, and a figure on one screen would not match the same figure on another. That is the sort of thing that makes people distrust a system without being able to say why.
And there was a third. A script exists whose whole job is to check that a tenant’s dashboard figures are sane before anyone demonstrates them. It contained the same logic, copied word for word. So it looked at the one per cent organisation, applied the same guess, saw a hundred per cent and passed.
What we recommended
The findings and the following rules are what we set out. The source for this article does not record the changes as shipped, so this write-up does not claim they are.
Put the unit in the storage. A percentage column named as a percentage and limited to between 0 and 100, or a fraction limited to between 0 and 1, leaves nothing to infer and turns a wrong unit into an error at the moment it is written.
Never rescale by looking at how big the number is. And when a value cannot be interpreted, refuse to show it. An empty tile saying “figure unavailable” is honest, while a tile showing a rescaled number is a claim.
The checking script should be independent of the thing it checks. The cheap version is one rule that needs no knowledge of the unit: a compliance rate above a hundred is a defect either way.
What it did not fix
Nothing about this was visible on screen. It was found by reading the code, and we have no way to find the next one of these except by reading the code.
The pattern, for anyone who reads a dashboard
For any percentage you rely on, ask in which unit it is stored and what stops the wrong unit being written. Ask for the worst site in your estate to be shown to you: if it looks perfect, that is the reason to look closer. And ask whether the test that checks the dashboard uses the same calculation as the dashboard. If so, it is a second copy, not a check.
Where this ends up
A compliance figure is worth putting on a screen only if its unit is stated rather than inferred, because in Sazinga Comply that figure is what tells an organisation what is due, what is evidenced and what has expired.