Field app slow or not working, but nothing is down
Reps could not start their shift though nothing was down. A slow Google lookup sat inside the request that had to succeed. How to check yours.
If your field team says the app is slow or not working and nothing on your side is down, look at what the slowest request depends on. Often a call to a third-party service sits inside the one action that has to succeed, such as starting a shift, so the whole action is only as reliable as the slowest outside service. The first check is to ask what, in that request, you do not control, and what the person sees when it is slow.
Here is how that looked for a field-sales team in rural India, and what fixing it took.
A field-sales operation with staff spread across rural India kept hearing the same complaint: “Could not start shift. Please try again.” Reps tapped the button, got an error, tapped again, and eventually got in. Every day, from different people, in different places.
The business cost was not the annoyance. It was that attendance had stopped being evidence. Shift records drive who was working, where they were, and whether a territory was covered at all — and nobody could now say whether a gap in the data meant a rep had not worked or the app had failed them.
The account was healthy the whole time
The first thing we checked was whether the backend was falling over. It was not. One of the reps generating complaints had 653 successful shifts over two years, almost daily. There was nothing wrong with that account, that device, or that user.
Then we counted the duplicates, and the shape of the problem appeared:
- 313 user-days in thirty days had more than one “start shift” row.
- Some users had five, milliseconds apart — one had three within eight milliseconds.
- Of 18,162 start rows in thirty days, 7,059 had no end.
Eight milliseconds is not a person changing their mind. That is one tap arriving on the server more than once, or a person tapping again because the first tap appeared to fail.
What was actually happening
The request was reaching the server and writing the row. The response was getting lost on the way back to a phone on a rural connection. The app saw a timeout, showed the error, and the rep tapped again — creating a second real shift for a shift that had already started.
Underneath that were two specific causes.
A third-party dependency inside the critical path. Starting a shift captured the rep’s location, and the code called Google’s geocoding API and waited for it before writing the attendance row. So whenever Google was slow — throttling, or simply hard to reach from a village — the whole request stalled until the phone gave up. Google was sitting inside the one operation that had to succeed for a person to start work.
No constraint to catch the retry. The code checked whether a shift already existed and then inserted one. Two taps arriving together both pass the check and both insert. The duplicates were not a bug in the sense of a mistake in a line of code; they were possible by design.
What we changed
The row is written first, immediately. The location lookup happens afterwards and nobody waits for it — if it fails, the shift is unaffected and the location is simply filled in late or not at all. We also moved off Google to OpenStreetMap, which is free, because once a lookup is not allowed to block anything, paying for it stops making sense.
Then the database was given the rule the code had been trying to enforce: a partial unique index allowing at most one active shift per person per day, in Indian local time. Concurrent taps can no longer both succeed. The second one now returns “shift already started”, which is the truth, instead of an error that invites a third attempt.
Cleaning up behind it removed 239 existing duplicates.
What it cost to leave the numbers coarser
This trade is worth stating because it went the other way. OpenStreetMap’s rural coverage is less precise than Google’s. For one rep’s location Google names the village; OpenStreetMap gets as far as the nearest town.
We took that deliberately. Location is now decorative — useful context on a record, never a reason a shift fails. Precision that can stall a person’s working day is worth less than a coarser label that cannot. If exact village names ever matter for a specific report, the provider is one setting.
How to check your own app
The dependency that hurts you is rarely the one you evaluated. Nobody chose to put an external API in the path of starting work; a location lookup was added next to a database write, and the two quietly became one operation that could only be as reliable as the slower half.
Two questions find most of these before your users do. What in this request can I not control, and what happens to the user when it is slow? And if the same tap arrives twice, what does the database do about it? The second one has to be answered by a constraint. Code that checks before it writes is not an answer, because between the check and the write is exactly where the second tap lands.
Where this ends up
The shift and attendance records described here belong to Sazinga Field, where a rep starts the day, takes orders on a phone and has the location stamped on the record. The rule that stops a second tap creating a second shift lives in its database rather than in its code, which is the only place it can be enforced.