Only a server that answered should spend a retry
A field rep wrote orders during a deployment window. The API was returning gateway errors for about a minute while it restarted. The rep came back to a queue full of writes marked as rejected, and nobody had ever seen a rejection.
The queue’s drain routine ran when the device’s connectivity state changed, and treated any response that was neither a network failure nor a conflict as final. So a temporary gateway error and a genuine validation rejection produced the identical outcome: a dead entry in the queue, with copy telling the user the office had refused their change. One of those is true and one of those is a lie, and the queue had no way to tell them apart.
There was no retry schedule at all. Not a badly tuned one — none. A write that failed once for any server-side reason stayed failed until a person went looking for it.
Two categories, and the line between them
The fix is a classification every offline queue needs and many do not have: is this failure transient or terminal?
Transient. The request did not get a verdict. Rate limiting, gateway and server errors, request timeouts, and any transport-level failure where nothing came back. The correct response is to try again later, because the write is probably fine and the world was busy.
Terminal. The server understood the request and refused it. Bad request, unauthenticated, forbidden, not found, unprocessable. Retrying is pointless — the same request will be refused the same way in an hour — and retrying it looks to the operator like a client that will not take no for an answer.
Conflicts sit outside both, because a conflict on a replayed write is ambiguous and needs its own classifier to decide whether the write already landed.
Where a rate-limit response carries an explicit retry-after header, that is honoured rather than overridden by the local schedule. The server has told you when it will be ready; guessing something different is not caution, it is ignoring an answer you already have.
The flaw in my own design, found before it shipped
The first implementation gave each queued write a budget of five attempts. Exceed it and the write is parked as failed for a human to deal with. Reasonable, and it counted every attempt including the ones where the request never reached anything.
A phone out of coverage for a day burns five attempts against thin air and parks a perfectly valid order as rejected. That is exactly the silent loss the offline contract exists to prevent, produced by the mechanism written to prevent it.
The correction is a single rule and it is the title of this piece: only a server that answered spends an attempt. A transport failure — no route, no response, connection dropped — reschedules without consuming budget. A phone can be off the network for a week and its queue is intact when it comes back. A server returning gateway errors forever will still eventually exhaust the budget and surface for a human, which is right, because that one is a real problem somebody needs to see.
The regression check asserts both halves. Transport failures do not decrement the budget; answered failures do. Only checking the first half passes on a queue that never gives up on anything, and that is its own kind of broken.
I would not have found this by testing. It came from asking what the mechanism does to the person it was written for, on the worst realistic day they have. That question is worth asking of any retry policy, any lockout, any rate limit and any expiry, and it is cheaper than the incident.
Jitter is not a detail
The retry delay grows exponentially, and it is randomised across the whole interval rather than having a small random amount added to a fixed value.
The reason is the shape of the deployment. Field staff are clustered — a branch office, a depot, a van park — and they come back onto the network together when the connection returns or the shift starts. With a deterministic schedule, every device that failed at the same moment retries at the same moment, and keeps doing so at every subsequent interval. The queue has manufactured a synchronised load spike against a server that was already struggling enough to cause the failure.
Full randomisation across the interval spreads them. It costs one line and it is the difference between a recovery and a second outage.
A third state, honestly labelled
The queue originally had two visible states: pending and failed. A write waiting on a retry is neither. Showing it as failed tells the user their order was rejected when it is about to succeed on its own; showing it as pending tells them nothing is wrong when something is.
So there is a third state — retrying — with its own copy, its own count on the status bar, and its own explanation of what will happen next. A user who can see that a write is retrying does not report it, and does not re-enter the order, which is the actual failure this prevents. Duplicate orders from users who thought the first one was lost are more expensive than the original problem.
That connects to the underlying rule for this whole subsystem: a queued write may never be silently dropped, and a state the user cannot distinguish is a silent drop with a label on it. Pending, retrying, needs-attention and rejected are four different situations, and each of them implies a different action from the person holding the phone.
Two implementation notes worth stealing
Adding the next-attempt time meant a schema change to a local database on devices already in use. That is a migration on hardware you do not control and cannot roll back. It is guarded by inspecting the existing table structure before altering it, so an app that upgrades from any earlier version converges, and one that has already been upgraded does not fail on a second run. A local schema change without that guard turns a routine release into a support queue.
And a schedule needs something to drive it. The original drain fired only on a connectivity change, which is fine for “we are back online” and useless for “try again in ninety seconds”. The drain now re-arms a timer for the earliest pending attempt, so the schedule is actually executed rather than merely recorded. A backoff policy that nothing invokes is a data model, not a behaviour.
Rules
Split failures into transient and terminal, and treat the ambiguous ones separately. No verdict means try again. A verdict means stop.
Only spend a retry attempt when a server answered. No answer is not a rejection, and treating it as one loses work that was never wrong.
Randomise the delay across the whole interval. Your users come back online in groups.
Give a self-healing write its own visible state. Users who cannot tell retrying from rejected will re-enter the work.
Guard local schema migrations against being applied twice. You do not control when the upgrade happens or from which version.