Let's talk
engineering

The same extension was charged twice, because a mirrored total has no memory

At two in the afternoon a counter clerk extended a rental by three days and entered the charge by hand: 1,593 rupees. At ten to four the same booking re-priced itself from the upstream rental system, which had also had the extension entered into it, and the ledger’s opening entry rose by exactly 1,593.

Both charges stand. The customer paid 8,732 — precisely the re-priced total, so he was billed correctly by the machine that took his money. Our ledger says he still owes 1,593, and it will hold that much back from his 5,000 rupee deposit when somebody settles the booking. He gets 3,407 back instead of the whole thing.

Nothing threw. Nothing was flagged. The booking looked exactly like a booking with an unpaid balance, which is a thing that happens constantly.

Where the mirrored number comes from

Every booking opens its ledger with a first-entry row carrying the total price the upstream system reports. That is a reasonable design: the upstream system is where the booking is made and where the rental is priced, and our settlement has to start from its number.

The problem is the word force. On every re-sync, the first entry is set to the upstream system’s current total. Not adjusted, not reconciled — set. There is no record anywhere of what it used to be, because the mirror is a value rather than a sequence of events.

Across the ledger’s first seven weeks, 97 of 546 bookings had their opening entry moved after creation. Most of those moves are legitimate — a booking is genuinely re-priced and the ledger should follow. The moves become dangerous only when somebody has also recorded a charge on our side for the same underlying change.

Two systems, both allowed to raise the same total

That is the actual shape of this bug, and it is not a coding error. It is two systems both holding authority over parts of one number, with no protocol between them.

An extension can be entered at our counter, which writes a ledger row. It can also be entered upstream, which raises the total and therefore raises our opening entry on the next sync. Both are legitimate ways for the office to do the job, and the office does both, sometimes on the same booking within the same afternoon.

Where two systems can both change the same figure, you need to decide which one owns it — and if neither can own it outright, every change has to carry enough provenance that the other system can recognise its own work. A total is not enough provenance. A total is the answer with the workings thrown away.

The audit found two more bookings sitting in exactly this state and, unusually, both of them balance to zero on screen so nothing looks wrong at all. On one, upstream raised the booking from 16,874 to 26,491 while a 3,481 extension had been added by hand. On the other, upstream lowered the booking from 72,806 to 64,310 while an 11,151 extension was added by hand. Neither can be resolved from our side alone — the question is whether the extension is already inside the re-priced total, and only the upstream system knows.

That is the real cost. Not the money, which is recoverable, but that the answer is unavailable from our records at all. When somebody eventually settles those two bookings, they will have to open another system and read it.

The second-order version of the same trap

The same mechanism nearly caused a much larger error on a different booking, and this is the version worth watching for.

That booking had 7,238 rupees of extension charges entered manually over several days. The upstream system still carried the original, un-extended total, because the extensions had never been entered there. A force-sync of the opening entry would have re-priced the booking down to the un-extended figure and then the manual rows would have been added on top — the mirror overwriting itself against history and the extension counted twice in the other direction.

A synchronisation that overwrites is safe only while it is the sole author. The moment anything else can write to the same field, an overwrite is a lost update with a timestamp.

What a fix would actually look like

The honest position is that this one was documented rather than fixed, because the fix is a design decision the client has to make and not something to smuggle in as a bug fix. But the options are worth stating because they generalise.

The first is to give the mirror an identity. The opening entry becomes a sequence of entries — an original and a set of re-price adjustments, each recording the previous value and the new one — so the ledger can be asked “did the upstream total move by 1,593 within an hour of a manual 1,593 extension” and answer it. That question is currently unanswerable without a manual comparison.

The second is to stop mirroring and start reconciling. On each sync, compare the upstream total against the sum of our own rental-bucket entries, and where they disagree, write the difference as its own entry with a reason. The number matches either way; the difference is that one of them leaves a trail.

The third is to make manual extensions the only route, or upstream extensions the only route, and enforce it. That is a process change, not a code change, and it is the cheapest of the three if the office will wear it.

The reason all of this was still recoverable

One fact made the whole audit a tidy exercise rather than a crisis: nothing was locked. Across the entire history of the ledger, no invoice had ever been finalised and exactly one payout had ever run. Every correction was still open.

That is luck as much as design — the refund pipeline had never completed once, for reasons that are their own story — but it is worth extracting the principle anyway. Between the point where money is calculated and the point where it becomes irreversible, there should be a step somebody has to take deliberately. Finalising an invoice is that step here. Everything before it can be corrected with a database update and a note. Everything after it is a refund, a phone call and an apology.

The other thing to take from this: an audit of every row since a subsystem went live cost two days and found fifteen bookings needing attention out of 547. Do it once, early, while the population is small enough that reading every row is possible. Nobody audits 50,000 bookings by hand, and by then the same defects are still there.

Working on something like this?

We build this kind of software, and we staff the teams that do.

Get in touch