The backup job failed silently for four days and four separate things were wrong

A nightly database backup produced zero objects for four days. Nobody knew, because a job that fails silently and a job that succeeds look identical from every direction except the one nobody checks.

When it was examined, there were four independent causes stacked on top of one another:

The dump command was prompting for a password. Running non-interactively, it failed immediately with an authentication error. On its own, that would be the whole bug.

The instance had no permission to write to the storage bucket. So even if the dump had produced a file, the upload would have failed. This second failure was completely hidden by the first — you cannot discover an upload permission problem while nothing is being uploaded.

The lifecycle rule that was supposed to expire old backups had never been applied. Which would have mattered later, in the form of a storage bill and eventually a quota.

A cleanup bug leaked disk. The script blanked the dump file variable on success, which disabled the cleanup trap that used it, leaving roughly 150 MB behind every day.

And the schedule was set in the wrong timezone — configured for two in the morning, running at half past seven local time, in the middle of the working day.

The stacking is the point

Four defects, and only one of them could be observed at a time. Fix the password prompt and you discover the permissions problem. Fix that and you find the lifecycle rule was never applied. Each fix reveals the next.

This is characteristic of any pipeline that has never successfully run end to end. A job that has never worked does not have a bug; it has a queue of bugs, and you can only see the first one. The implication for scheduling work is that you should not estimate “fix the backup” as one task. You should expect to iterate until a complete, verified artefact exists at the far end.

Which leads directly to the only thing that actually matters here.

The real defect was the absence of alerting

The write-up named the top remaining gap in one sentence: the job failed silently for four days.

That is the actual defect. The four technical faults are ordinary; every one of them would have been fixed within an hour of being noticed. The system-level failure is that four days passed with no backups and nothing said anything.

A scheduled job needs three things, and most have only the first:

1. It runs. Cron entry, timer unit, scheduler. Everyone has this.
2. It reports failure. Non-zero exit produces an alert somewhere a person will see.
3. It reports silence. The absence of a success signal within an expected window produces an alert.

The third is the one that catches this class of failure, and it is the one almost nobody builds. A job that fails loudly is a solved problem. A job that never runs at all — because the timer was disabled, or the machine was rebuilt, or the scheduler entry was lost in a migration — produces no failure to report. It produces nothing, and nothing is exactly what a healthy quiet system also produces.

The test to apply: if this job silently stopped existing, how long until somebody found out? If the answer is “when we need it”, you do not have a backup, you have an intention.

The renewal that had been dead for months

A related failure from another system, and it is a better story because of how it was hidden.

Automated certificate renewal had not run since early June. The systemd timer that drives it was masked — symlinked to the null device, so it could never start. Meanwhile the package had installed a cron entry that looked exactly like a safety net, sitting there in the crontab, appearing to be the mechanism. That entry self-disables when systemd is present, which it was.

So: renewal was entirely off, with a cron entry looking like cover.

This was found ten days before four certificates would have expired. Unmasking the timer renewed all four in a single run — which is proof that it works rather than an expectation that it will.

The general shape is the same as the decorative security controls I have written about before, and it is the most common form of infrastructure failure I encounter: a mechanism that is present, plausible, and inert. The cron entry existed. The timer existed. Neither was doing anything, and the presence of two of them made it less likely anyone would check, because redundancy reads as safety.

Monitoring the monitor

One decision from the same work is worth carrying, because it is the difference between monitoring that works and monitoring that reassures.

A checker running on the monitored server cannot report that the server is down. If the machine is unreachable, so is the thing watching it. Silence and health become indistinguishable, which is the same failure as the missing backup alert, one level up.

Three further rules came out of an actual outage:

Transport-level checks are not enough. During the incident, the port answered, returning a redirect in under half a second, and DNS resolved perfectly. Only a full request to the application revealed the gateway error. A TCP connect proves a socket is listening, which is not the same as the application working.

Assert on the status code and on a known string in the body. A successful status carrying an error page or a blank shell should count as down. This matters especially with single-page applications, where the server returns success for every path — a catch-all response means probing a URL proves nothing about whether the page exists.

Treat very slow as down. A response taking ten seconds against a healthy baseline of around one second is a failure from the user’s perspective, and it is usually the leading indicator of the outage rather than a separate condition.

The rule

Verify infrastructure by its output, not by its configuration.

The backup was configured. The timer was installed. The cron entry was present. Every one of those facts was true and none of them meant anything, because the only question that matters is whether there is a usable artefact at the end, produced within the window you expect, with something that complains when there is not.

Go and look at the most recent backup for a system you are responsible for. Check its timestamp and its size. If you cannot do that in under a minute, that is the finding.