Five AI providers configured, half the requests failed, and only one was ever tried

A feature that generates quiz questions from a piece of written content was failing about half the time. The system had five different model providers configured, each with its own key, arranged in a rotation so that if one was rate-limited or down the request would move to the next. With five providers, an outright failure should have been rare.

The first hypothesis was the keys. Someone’s billing had lapsed, or a key had been revoked, or a free tier had run out. Reasonable, and wrong. The second hypothesis was the response parser — models return awkward JSON, and a parser that trips on a trailing comma would produce exactly this kind of intermittent failure. Also reasonable, and also wrong.

What settled it was three lines of log output next to each other:

provider A -> 402 Payment Required
provider B -> 200 OK
client     -> AI_EMPTY_REPLY

Provider B returned 200. The rotation stopped there, because that is what a rotation does when a call succeeds. Provider B’s 200 contained an empty body. Providers C, D and E — three working keys with credit on them — were never contacted.

Where the logic was wrong

The rotation loop advanced on transport-level failure: a 429 meant back off and try the next key, a 5xx meant try the next key, a connection error meant try the next key. Anything else was treated as a result, and the loop returned.

The emptiness check happened after the loop had exited. The code took whatever the loop returned, stripped it, found nothing there, and raised an error. By that point the loop was gone and with it any possibility of trying another provider.

So the failure was not in either place anyone looked. It was in the boundary between them — a correct retry loop and a correct validity check, arranged so that the check could not inform the retry. Each piece was defensible on its own. Together they meant that a provider returning successful nothing terminated a chain of four working alternatives.

The fix was small and structural: raise a provider-level error inside the loop when a reply is unusable, so an unusable 200 advances the rotation exactly as a 429 does. The quiz path passes an additional predicate, because for that route a reply can be non-empty and still unusable — prose where structured data was required is, for this purpose, the same failure as no reply at all.

The general shape

A transport-level success is not an application-level success, and any retry policy written against transport codes alone will eventually stop on a response that satisfies nobody.

This is not specific to language models, though they make it vivid because an empty 200 is a routine occurrence rather than an anomaly. The same structure appears in a payment gateway returning 200 with a body saying the charge was declined, an upstream service returning an empty array when it means “I could not reach my own database”, and a cached response served with a success status containing a stale error page.

The question to ask of any fallback chain is not “does it retry on failure” but “what is this chain’s definition of success, and is it the same as the caller’s?” If the loop’s definition is status < 400 and the caller’s definition is parses into a valid object with at least one question, the gap between them is where requests go to die.

Put the caller’s validity check inside the loop. That is the whole lesson, and it costs nothing at the time you build it.

Two related failures from the same subsystem

Once the rotation actually rotated, two further problems surfaced that had been masked by it.

A dead key stayed at the front of the queue forever. One provider was returning 402 on every call. There was an endpoint to test a key, and it correctly reported the key as failing — but nothing persisted that result anywhere, and the list endpoint only ever reported whether a key was enabled. Meanwhile the auto-disable rule covered 401 and 403 and did not include 402. So a billing-dead key sat permanently at the head of a rotation ordered by least-recently-used, burning two round trips on every single request before anything useful happened.

The fix was to persist the outcome of each test — when it last ran, whether it passed, what the error was — and to treat a billing failure as its own class that disables the key with a stated reason. A key’s health is state worth storing. Testing it and throwing the answer away is a check that produces no information.

Then the timeout stopped making sense. The mobile client applied one flat 20-second timeout to every request and mapped an abort to a network error. That is a fine default for a call to your own database. It is a poor default for a call that waits on somebody else’s model, and it is a terrible default for a call that may wait on several models in sequence.

Measured on a real rewrite request: first provider skipped, second answered in 0.7 seconds with an unusable model, third took 5.6 seconds before reporting rate-limiting, fourth answered properly at 15.2 seconds. Total elapsed: 21.6 seconds, against a 20.0 second abort.

The work completed. The token cost was incurred. The user saw “no connection”, which was a lie in both directions — the network was fine and the answer existed. And because the error was categorised as a network error, the natural next step for anyone debugging it was to look at connectivity, which is exactly where the answer was not.

The fix was a per-route timeout budget: a long budget for the small number of routes that wait on an external model, the ordinary short one everywhere else, and a distinct timeout error type so that “we gave up waiting” is never again reported as “we could not reach the server”.

What ties these together

All three failures share a structure worth naming. In each case a piece of the system had a local definition of failure that was narrower than the global one.

The rotation’s definition of failure did not include a valid-looking empty response. The key store’s definition of a bad key did not include one that was out of credit. The client’s definition of a reasonable wait did not account for its own retry chain running behind the request.

None of these are bugs in the ordinary sense. Nothing was mistyped and nothing crashed. They are seams — places where two correct components meet and neither is responsible for the space between them. Seams do not show up in unit tests, because unit tests exercise components. They show up in production as an intermittent failure rate that nobody can reproduce, which is precisely how this one presented.

Two habits help. First, when a system has a retry policy, write down what it advances on and what it treats as final, as a list, and check that list against every failure the downstream code knows how to raise. Second, when debugging an intermittent failure, log the sequence rather than the outcome. The bug here was invisible in the error and obvious in three consecutive log lines, and that is a very common ratio.

The transferable rule: decide what “success” means once, and make sure every layer that can retry is using that definition rather than its own.