Paid once, credited twice, then asked for the deposit again

A customer pays by card through the link you sent. Your system records the payment twice, so the account shows double the money that arrived. Then, having paid the deposit, the customer is asked for the deposit again, and the amount your system thinks is owed has quietly doubled too.

That was the first genuine card payment through a new payment provider on a car rental system. One payment, one customer, two separate mistakes in the same transaction. Both would have looked fine on every screen until somebody tried to reconcile the bank statement or the customer rang to ask why they were being charged again.

The cost of the first mistake is a wrong figure in the accounts and a wrong amount the customer is told they have paid. The cost of the second is an awkward conversation and, in the same system, a sister bug had already told drivers to collect nothing on six live bookings. None of these cost anything to build correctly from the start. All of them cost a customer’s money to learn afterwards.

What was actually going on

The payment provider sends two messages for one payment made through a link: one saying the payment was taken, one saying the link was paid. They describe the same money. They arrived 434 milliseconds apart.

The system had a guard. Before recording a payment it checked whether the booking was already marked paid, and only wrote the credit if not. Both messages checked while the booking still said unpaid. Both passed. Both wrote. A check followed by a write is two steps with a gap between them, and in the gap the other message did the same thing. The code reads as a guard, every line of it is correct, and the failure is entirely in the assumption that nothing runs in between.

The second mistake was different in kind. When a booking expects a deposit, the system writes a record for it up front, marked pending. That record is how the system knows money is owed. The code that recorded the deposit payment wrote a new record beside it and left the pending one standing. So the customer had paid, the pending record still said money was owed, and the expected deposit total had doubled.

What we changed

For the double credit: the check and the write are now one step. The system no longer asks “is this paid?” and then writes; it tells the database “mark this paid, but only if it is not already”, and the database reports whether that took. One message wins, the other does nothing. The database is already the referee for two things changing one record at once; the fix is to let it referee rather than deciding in code, where there is no such guarantee.

For the doubled deposit: paying a deposit now settles the pending record in place, oldest first, with part payments splitting a record so the halves still add up to the original. Fulfilling an expectation means changing the record that holds it, not adding a second one next to it.

And one rule that paid for itself within days: a message from the provider is a notification, not proof. The system records the raw message and replies “received”, then separately asks the provider whether the payment really exists before crediting anything. Shortly after that was built, a test message from the provider’s own dashboard arrived, correctly signed, for a substantial sum. Asked about it, the provider said the transaction did not exist. A system that trusted the message would have credited a booking for money that never moved.

What it did not fix

The double-credit fix was never provider-specific, and it is worth saying why. It is easy to file this as “that provider sends duplicates”, fix it for them, and move on. But a payment message arriving while a reconciliation sweep is mid-flight does exactly the same thing, and that has nothing to do with any provider. Any system with more than one path that can record the same fact has this race, whether or not anything external duplicates anything.

There is also a hard reason the “does this payment exist?” question cannot be asked while the message is being received: the provider expects an answer within five seconds and treats anything slower as a failed delivery, which turns a successful payment into a storm of repeats. So the shape is receive, store, acknowledge, and interpret later.

The pattern behind both

Both mistakes were values that look like evidence and are not. A pending deposit record was counted as a deposit held — that is the bug that told drivers to collect nothing on six live bookings. A field from the booking platform holding the advance amount was treated as proof the advance had been paid; it is the amount expected, and it is present on unpaid bookings too. A timestamp called “updated at” was used to decide when a job was completed, when in fact it was only ever set once, on creation, so every record showed its creation time and nobody looked. A provider’s message was treated as proof that money moved.

Each is present, plausible, and answers a different question from the one being asked of it. The habit that catches them: for any field you are about to rely on, state in one sentence what it means and who writes it. If the sentence has an “or” in it — “it means the advance was paid, or that an advance was expected” — you have found one.

The mechanism

The double-credit fix is a conditional update whose row count is the verdict:

UPDATE payments SET status = 'paid', ... WHERE id = :id AND status <> 'paid'

One row updated means you won and own the credit. Zero means somebody got there first. That generalises to almost every check-then-act against a database: reserving stock, claiming a job from a queue, assigning a sequence number, marking something processed.

The deposit fix settles pending rows under a row lock, oldest first. The provider rule is: verify the signature, persist the raw payload, return success, and confirm against the provider’s status API out of band before touching the ledger. The raw event is a fact you can always re-process; an interpretation you got wrong is a fact you have destroyed.

Where this ends up

All three rules sit in the payment path of Sazinga Rentals, where a provider can send more than one message for the same money and the booking ledger has to credit it once.