The field team said the app was broken. The app was fine.

A field-sales operation with staff spread across rural India kept hearing the same complaint: “Could not start shift. Please try again.” Reps tapped the button, got an error, tapped again, and eventually got in. Every day, from different people, in different places.

The business cost was not the annoyance. It was that attendance had stopped being evidence. Shift records drive who was working, where they were, and whether a territory was covered at all — and nobody could now say whether a gap in the data meant a rep had not worked or the app had failed them.

The account was healthy the whole time

The first thing we checked was whether the backend was falling over. It was not. One of the reps generating complaints had 653 successful shifts over two years, almost daily. There was nothing wrong with that account, that device, or that user.

Then we counted the duplicates, and the shape of the problem appeared:

• 313 user-days in thirty days had more than one “start shift” row.
• Some users had five, milliseconds apart — one had three within eight milliseconds.
• Of 18,162 start rows in thirty days, 7,059 had no end.

Eight milliseconds is not a person changing their mind. That is one tap arriving on the server more than once, or a person tapping again because the first tap appeared to fail.

What was actually happening

The request was reaching the server and writing the row. The response was getting lost on the way back to a phone on a rural connection. The app saw a timeout, showed the error, and the rep tapped again — creating a second real shift for a shift that had already started.

Underneath that were two specific causes.

A third-party dependency inside the critical path. Starting a shift captured the rep’s location, and the code called Google’s geocoding API and waited for it before writing the attendance row. So whenever Google was slow — throttling, or simply hard to reach from a village — the whole request stalled until the phone gave up. Google was sitting inside the one operation that had to succeed for a person to start work.

No constraint to catch the retry. The code checked whether a shift already existed and then inserted one. Two taps arriving together both pass the check and both insert. The duplicates were not a bug in the sense of a mistake in a line of code; they were possible by design.

What we changed

The row is written first, immediately. The location lookup happens afterwards and nobody waits for it — if it fails, the shift is unaffected and the location is simply filled in late or not at all. We also moved off Google to OpenStreetMap, which is free, because once a lookup is not allowed to block anything, paying for it stops making sense.

Then the database was given the rule the code had been trying to enforce: a partial unique index allowing at most one active shift per person per day, in Indian local time. Concurrent taps can no longer both succeed. The second one now returns “shift already started”, which is the truth, instead of an error that invites a third attempt.

Cleaning up behind it removed 239 existing duplicates.

What it cost to leave the numbers coarser

This trade is worth stating because it went the other way. OpenStreetMap’s rural coverage is less precise than Google’s. For one rep’s location Google names the village; OpenStreetMap gets as far as the nearest town.

We took that deliberately. Location is now decorative — useful context on a record, never a reason a shift fails. Precision that can stall a person’s working day is worth less than a coarser label that cannot. If exact village names ever matter for a specific report, the provider is one setting.

The general version

The dependency that hurts you is rarely the one you evaluated. Nobody chose to put an external API in the path of starting work; a location lookup was added next to a database write, and the two quietly became one operation that could only be as reliable as the slower half.

Two questions find most of these before your users do. What in this request can I not control, and what happens to the user when it is slow? And if the same tap arrives twice, what does the database do about it? The second one has to be answered by a constraint. Code that checks before it writes is not an answer, because between the check and the write is exactly where the second tap lands.