Five visitors took the site down for four hours, and the database was idle throughout

A booking site went down and stayed down for four hours. Every request timed out at the gateway. The obvious reading is load: something spiked, the server could not keep up.

The measurements said otherwise, and they said it clearly.

Traffic was roughly twenty requests per minute. System load average was 0.15. The database was entirely idle — every connection sitting in a sleep state, zero active queries. Nothing was working hard. Nothing was working at all.

The application workers were all sitting in a polling state, each holding an established network connection to the site’s own public address.

The mechanism

The front end of the application called its own back end by public hostname. A request that had already been accepted by the server went back out through the CDN, resolved the public address, came back to the same server, and was accepted as a new request.

Which means every such page view occupied two worker processes: one for the original request, and one for the internal call it made to itself.

The worker pool was configured for five children. Five simultaneous visitors occupied all five workers, each of them waiting on a request that needed a worker that would never become free. The pool deadlocked, and — this is the part that makes it four hours rather than four seconds — it could not recover on its own, because no worker could ever finish.

The reason it was permanent rather than merely slow was a second defect: the self-call had its timeout set to zero, which in that HTTP client means no limit at all. A stalled self-call blocked forever. Two of the workers had been blocked for fifteen to eighteen hours before the outage even became visible, which is why it took only three more visitors to finish the job.

And there was a third, bitter detail. The calling code already degraded gracefully — on a connection failure it falls back to a cached copy, even an expired one. That fallback is good engineering and it never got to run, because the call never failed. It hung. Graceful degradation only helps you if the failure mode you designed for is the one you get, and “hangs indefinitely” is not “fails”.

Three defects had to line up: the self-call, the unlimited timeout, and a small worker pool. Fixing any one of them breaks the deadlock. That is typical of outages that last hours — single defects produce errors, and it takes a conjunction to produce a system that cannot recover.

What the diagnosis actually required

The useful part of this story is not the bug, it is how it was separated from the thing it looked like.

An overloaded server and a deadlocked server present identically from the outside: requests time out, the gateway returns errors, users complain. The instinct is to restart, and then to add capacity. Restarting clears it for a few minutes. Adding capacity raises the number of simultaneous visitors required to deadlock it from five to ten, which delays the next outage and guarantees it.

Four measurements separated the two cases, and each of them is a check worth having in your head:

Is the database busy? If the application is overloaded, the database is usually working. Every connection idle with zero active queries means the application is not asking it for anything, which means the application is not doing work — it is waiting.

What is the load average? A saturated server has a high load average. A blocked one has a low one. Blocked processes do not consume CPU. A server that is “down” with a load average of 0.15 is not out of capacity.

What are the workers actually doing? Inspecting the worker processes showed them all in a polling state holding open sockets, and — decisively — the remote address on those sockets was the machine’s own public address. That single observation is the whole diagnosis. A worker waiting on its own server is not a load problem in any sense.

Was there a warning? The worker pool log announced that it had reached its maximum children about a minute before the first gateway timeout. That message was sitting there the whole time. It is the kind of log line that appears routinely under normal load and gets ignored, which is precisely why it was ignored here.

Why applications call themselves

This is not an unusual mistake, and it is worth understanding why it happens rather than treating it as carelessness.

The application needs data from its own API. The API has a public URL. The public URL is in a configuration file, already, because the browser needs it. Using it from the server side is one line and works perfectly in development, where the developer’s machine is not the server. It works in staging. It works in production too, right up until concurrency reaches the size of the worker pool.

The correct forms are to call the internal handler directly in-process, or to call the loopback address, or — if it genuinely must be an HTTP call — to configure a separate internal base URL that does not leave the machine.

The general rule: a server should never make a network request that resolves back to itself. If your architecture has a component calling its own public hostname, the concurrency limit of that component is now half of what you think it is, and the failure mode when you reach it is a deadlock rather than a slowdown.

There is a related version of this on any request-handling process that makes synchronous outbound calls: every synchronous outbound call halves your effective concurrency for the duration of that call. Self-calls are the pathological case because the resource you are waiting for is the resource you are holding.

Set a timeout on everything

The second defect deserves its own paragraph, because it is the one that turned a temporary degradation into an unrecoverable one.

Every outbound network call needs a timeout, and the default in most HTTP clients is either none or something absurdly long. “No limit” is never the right value in a request handler. A call that has not answered in a few seconds is not going to answer usefully; the user has gone. Failing fast keeps the worker available, and — as this incident demonstrated — the fallback path you already wrote starts working.

After the fix, the self-call resolved in 0.04 seconds instead of hanging, and ten concurrent requests all completed in about half a second. Five used to be enough to deadlock it.

The takeaway

When a system is unresponsive, establish whether it is busy or blocked before you do anything else. They look the same to a user and they have opposite remedies: a busy system needs more capacity, and a blocked system gets worse when you give it more, because you have raised the threshold rather than removed the cycle.

The distinguishing evidence is almost always available and almost always ignored: an idle database, a low load average, and processes waiting on something. Look at what the workers are waiting for. In a deadlock, one of them is waiting for something the system itself is holding, and once you see that, the rest of the diagnosis takes minutes.