The offline queue marked lost orders as delivered, because every 409 looked the same
A field sales rep places an order on a phone with no signal. The write goes into a local queue. When the connection returns, the queue drains and posts each write to the server. Ordinary offline-first design, and the whole point of it is that the rep never loses work.
The specification said the server returns 409 when a create arrives carrying a client-generated id that already exists, and the client treats that as “already applied” — mark it sent, move on. That is the correct idempotency contract, and it was written down.
Two things were wrong with it in practice, and together they produced exactly the failure the architecture existed to prevent.
Wrong thing one: the server did not return 409
A duplicate primary key raised a database integrity error, and that error was only caught inside one import routine. Everywhere else it propagated as an unhandled exception and the client received a 500.
So the sequence was: the rep’s order reaches the server and is written. The response is lost — the phone dropped back out of coverage, which is the normal condition for this application. The queue retries. The server rejects the duplicate id with a 500. The client, seeing something that is not 409, marks the write as failed.
The order exists in the database. The rep is looking at a screen saying it did not go through. That is the precise failure offline-first exists to prevent, produced by the mechanism intended to prevent it.
The fix was to map the database’s integrity violation codes centrally: a unique violation on the primary key becomes 409 with a code meaning this id already exists — an idempotent replay — and a unique violation on any other constraint becomes 409 with a different code meaning this value clashes with another row. A foreign key violation becomes its own conflict code.
One rule was applied deliberately here and it is worth stating: every other integrity class is re-raised so genuine bugs still surface as 500s. The temptation when writing a handler like this is to catch broadly and return something plausible. That launders real defects into believable conflicts, and you never hear about them again. A translation layer should translate the cases you have understood and let everything else through loudly.
Wrong thing two: the client believed every 409
Once the server was returning conflicts properly, a worse problem became visible. The queue’s drain logic treated any 409 as already-applied. It marked the write as sent and deleted the queued record.
The contract said the queue never silently drops a write. With blanket-409 handling, that was true for exactly one kind of 409 and false for all the rest.
Consider what else returns 409. A create where the business code clashes — two customers with the same reference, entered by two different reps. That is not a replay; that is a genuine rejection, and the rep’s work has just been deleted with no record of it. Or a state transition, say marking a dispatch as delivered, where the record has already moved past that state. Transitions carry no client-generated id at all, so “already applied” is not even a meaningful interpretation — the conflict means the operation did not happen.
The rewrite branches on the error code rather than the status:
- A conflict specifically meaning this id already exists is a replay. Mark it applied. This is the only case that can be settled from the response alone.
- Any other conflict on a create that carried a client id is unresolvable from the response, so the client asks. It issues a probe — fetch the record by its client id, or fetch the list that would contain it. A 200 means the write landed and this is a replay. A 404 means it did not.
- A conflict on an operation with no client id is not applied. There is nothing to probe and nothing to assume.
The principle: nothing is marked as sent unless it can be proved. Anything unprovable goes into a new explicit conflict state that surfaces to the user, rather than being resolved optimistically in either direction.
There was one subtlety that made the probe necessary rather than merely tidy. For customers and products, the server checks the business code before the primary key. So a genuine replay and a genuine clash both come back with the identical duplicate-code error. The error code cannot distinguish them. Only existence can — which is why the resolution has to be a question to the server rather than an inference from the answer it already gave.
The retry budget was spending itself on the wrong failures
One more defect from the same subsystem, found in the new code before it shipped, and it is the kind of thing that only shows up if you think about the field conditions rather than the code.
The queue had exponential backoff with a five-attempt budget. Transport failures — no connection at all — were consuming attempts. Which means a phone out of coverage for a working day would exhaust its budget on failures that carried no information whatsoever, and park a perfectly valid order in a failed state. The rep loses the order because they were somewhere with no signal, which is the entire scenario the queue was built for.
The rule now is that only a server that actually answered spends an attempt. A transport failure means try again later; it is not evidence about the request.
Alongside that, transient and terminal failures were separated. Before, a 503 from a restarting gateway and a 422 validation rejection produced the identical dead state — one of which will succeed if you wait thirty seconds and one of which will never succeed no matter how long you wait. Now 429, 5xx, 408 and transport errors retry, honouring a server-supplied retry delay where one is given, with full jitter so that a whole branch coming back online does not retry in the same instant. The 4xx family that indicates the request itself is wrong is terminal and surfaces immediately.
The contract questions
Offline sync is usually discussed as a feature — “the app works offline”. It is more useful to treat it as a contract between client and server, and to write the contract down before either side is built, because almost every bug above came from the two sides holding compatible-sounding but different beliefs.
The questions the contract has to answer:
- What does the client send that lets the server recognise a replay? A client-generated id on creates. Time-ordered ids rather than random ones, so server-side index inserts stay local rather than scattering across the index.
- Which operations accept that id and which reject it? Creates accept it. Transitions must reject it, because a transition is not idempotent by identity and pretending otherwise produces the false-applied case above.
- What exact response means “already applied”, and can it be confused with anything else? If two different situations produce the same code, the client needs a second question it can ask.
- Which operations are safe to queue at all? Some are not. A cancel replayed against a record that has since moved state would be misreported as applied — so it stays online-only. An upload depending on a short-lived signed URL and a local file reference cannot be replayed from a body-only queue, so it is not queued, and the interface says so before the user confirms rather than pretending an upload is pending.
- What does the client do with an answer it cannot classify? If the answer is “guess”, the queue will eventually lie to somebody.
That last question is the one that separates a queue that works from a queue that mostly works. Optimistic resolution of an ambiguous conflict is a decision to occasionally delete a user’s work silently — and silently is the operative word, because the user finds out days later when someone asks where the order went.
Design the failure classification before the happy path. The happy path in offline sync is trivial; every bit of the difficulty is in what the client concludes when the answer is unclear.