The first real card payment was credited twice, 434 milliseconds apart

The first genuine card payment through a new gateway integration was credited twice. One payment, recorded as double the amount.

The gateway sends two events for a payment made through a payment link: one saying the payment was captured, one saying the link was paid. They describe the same money. They arrived 434 milliseconds apart.

The handler was guarded. It read the payment record, checked whether the status was already marked as paid, and if not, wrote the credit. Both events read the row while it still said unpaid. Both passed the check. Both wrote.

Why the guard was not a guard

A read followed by a check followed by a write is three statements with gaps between them, and in those gaps anything can happen. This is the classic time-of-check to time-of-use race, and it is worth being precise about why it is so easy to write by accident: the code reads as a guard. It looks correct. Every line of it is correct. The failure is entirely in the assumption that nothing runs between them.

The fix was to make the claim the UPDATE itself. Rather than checking the status and then writing, the write carries the condition:

UPDATE payments SET status = 'paid', ... WHERE id = :id AND status <> 'paid'

Then look at the row count. One row updated means you won and you own the credit. Zero rows means someone else got there first, and you do nothing.

The database is already serialising concurrent updates to a row. All you have to do is let it decide the winner rather than deciding it yourself in application code where you have no such guarantee.

This generalises to almost every “check then act” pattern against a database. Reserving stock, claiming a job from a queue, assigning a sequence number, marking something as processed — all of them should be a conditional update whose row count tells you whether you won, not a select followed by an if.

The most important note in the write-up of this bug: this was never gateway-specific. It is easy to file a bug like this as “that provider sends duplicate webhooks”, fix it for that provider, and move on. But a webhook arriving while a reconciliation sweep is mid-flight does exactly the same thing, and that has nothing to do with any provider. Any system with more than one path that can record the same fact has this race, whether or not anything external duplicates anything.

The second bug in the same payment

The same transaction exposed a completely different defect, and this one is a data-modelling error rather than a concurrency error.

When a booking expects a security deposit, the system creates a ledger row for it up front, marked as pending. That row is the expectation — it is how the system knows money is owed.

The code that recorded a deposit payment wrote a new row alongside it and left the pending one standing. So the customer paid the deposit, was immediately asked for the deposit again, and the expected deposit total quietly doubled, because now there were two rows and one of them still said pending.

The fix was to settle the pending rows in place, oldest first, under a row lock, with part payments splitting a row so that the halves still sum to the original figure.

The general principle: when a record represents an expectation, fulfilling it means changing that record, not adding a second one next to it. Adding a row is the instinct — ledgers are append-only, after all — but the pending row is not a transaction, it is a claim about the future. Two rows where one is an expectation and one is a fulfilment, with no link between them, means every query has to know which is which, and one of them will not.

The pattern behind both

There is a theme running through this codebase that is worth naming, because once you have the name you start seeing it everywhere: values that look like evidence and are not.

• A ledger row marked pending was counted as a deposit held. It is a promise, not a payment. That one bug meant the driver app told drivers to collect nothing on six live bookings, and every screen looked entirely normal.
• A field from an upstream system holding the advance amount was treated as proof that the advance had been paid. It is the amount expected, and it is populated on unpaid bookings too.
• A timestamp column called “updated at” was used to determine when a job was completed. The model had timestamps disabled and the column had only a default value, so it was stamped once on insert and never moved again — meaning it recorded when the row was created, not when anything changed. Every row in the table had it equal to the creation time, which is exactly what you would see if you looked, and nobody looked.
• A webhook payload was treated as proof that money moved.

Each of these is a value that is present, plausible, and answering a different question from the one being asked of it. They are much harder to find than nulls or errors, because the system produces confident, well-formatted, entirely wrong output.

The habit that catches them: for any field you are about to rely on for a decision, state in one sentence what it actually means and who writes it. If you cannot, do not use it yet. If the sentence has an “or” in it — “it means the advance was paid, or that an advance was expected” — you have found one.

Never trust the webhook alone

One more rule from the same integration, which paid for itself immediately.

A webhook is a notification, not evidence. The handler verifies the signature, persists the raw payload, returns success — and then, out of band, calls the provider’s own status API to confirm the payment before writing anything to the ledger.

This is not defensive paranoia. Shortly after it was built, a test webhook from the gateway’s own dashboard arrived with a valid signature for a substantial amount. The signature verified correctly. The provider’s status API, asked about that transaction, replied that it did not exist. A system that trusted the payload would have credited a booking for money that never moved.

There is also a hard reason the verification cannot happen inside the handler: the gateway expects a successful response within five seconds and treats anything else as a delivery failure. Calling an external API inside that budget risks spending it on exactly the transaction you care about, and a timeout converts a successful payment into a retry storm.

So the shape is: receive, verify the signature, store the raw payload, return success. Interpret later. Store first and interpret second — the raw event is a fact you can always re-process, and an interpretation you got wrong is a fact you have destroyed.

The three rules

1. Make the claim the update. If two things could race, let the database pick the winner and read the row count.
2. Fulfil an expectation by changing its record, not by adding a second one beside it.
3. Store the event, then interpret it. A notification is not evidence, and evidence comes from asking the system of record.

None of these cost anything to build in from the start. All three of them cost a customer’s money to learn afterwards.