The deploy printed a warning, then printed Done, and the old code kept serving
Every deploy for weeks had ended with a warning about a missing privilege, and then the word Done. The files were copied. The build was fresh. The application service was never restarted, so the machine carried on serving the previous version of the code, and the deploy log said it had succeeded.
That is the dangerous shape of the bug, rather than the warning itself. A deploy that fails is an annoyance. A deploy that reports success without doing the thing it exists to do quietly detaches what you believe is running from what is running, and nothing tells you until a fix you shipped three days ago turns out not to be there.
Why the privilege was missing
The deploy script runs as a dedicated, unprivileged application user, so that the checked-out code stays owned by that user rather than by root. That is the correct choice and I would make it again.
The last step of the script restarts the service. Restarting a system service requires privilege that the application user does not have. The step attempted it, could not, warned, and continued.
A neighbouring project on the same machine has never hit this, because it has no dedicated deploy user and runs the same script as root, where the restart simply works. So the failure was specific to the project that had done the more careful thing.
A warning that does not change the exit status is a lie told by the tooling. If a step is required for the deploy to mean anything, it must fail the run. If it is genuinely optional, it should not be in the deploy. There is very little in between, and “warn and carry on” is where optional steps and required steps become indistinguishable.
Granting exactly one thing
The fix is a privilege grant, and privilege grants are where people get lazy under time pressure.
What was granted: that one user, the ability to run one restart, of one named service unit, at its absolute path, with those exact arguments. Nothing else.
Three things about doing it safely.
Validate before installing. The privilege configuration directory on that machine is shared by every project on the box, and a syntax error anywhere in it breaks the privilege system for all of them — including, potentially, your own ability to fix it. The file was checked with the validator before being put in place, which takes one command and removes the only genuinely catastrophic outcome.
Prove the blast radius rather than describing it. After installing, I confirmed that the same user still cannot stop the service and still cannot obtain a shell. A grant you have not tested the edges of is a grant whose edges you are guessing at.
Verify the effect, not the absence of the error. The warning disappearing proves the grant parsed. What proves the deploy works is that the service’s activation timestamp now matches the restart line in the deploy log. Those are different claims and only the second one is about the thing you cared about.
Config that was committed and never deployed
The same session turned up the sibling problem, which is subtler and more common.
Images were being served with no cache headers at all. The web server configuration block that sets them existed, was correct, and was committed to the repository — in the example configuration file.
The deploy script synchronises the web directory. It does not touch the web server configuration, because nothing in the pipeline ever has. So a block that had been written, reviewed and committed had never been live anywhere, and there was no signal of that beyond the missing header.
A configuration file in your repository that your deploy does not install is documentation, not configuration. Either the pipeline applies it or you should stop pretending it is deployed. The in-between state — an example file that someone is supposed to copy by hand — is where the running system and the repository drift apart without either looking wrong.
Two related landmines in the same pipeline are worth listing because they are all the same species.
The landing-page synchronisation uses a delete flag, so it removes anything in the destination not present in the source. Once a second application was installed into a subdirectory of that destination, the next deploy of the landing page would have deleted it. Found by reading the script rather than by watching it happen, which was luck.
And the web application’s production environment file was matched by a broad ignore rule in the repository root, so it was never committed. The server build therefore fell back to its development defaults: a loopback API address and an empty identity client id. That produces an application which asks each visitor’s own machine for the API — something no test catches, because every test runs on a machine where that address happens to work.
The fix there was two-part, and the second half is the general lesson. Committing the file solves the instance; it contains no secrets. Making the default follow the build mode — production builds default to the production address, development builds to localhost — means the environment variable becomes an override rather than the only thing standing between you and a broken deployment. Arrange your defaults so that the failure of your configuration produces the safe outcome, not the developer’s convenience.
What lives only on the box
The privilege file lives in a system directory. It is in no backup, it is in no repository, and a machine rebuild would silently lose it — reproducing exactly the original bug, months later, with nobody remembering why.
It is now committed to the repository, with a step in the first-deploy instructions and a row in the runbook.
The habit that follows: after any fix applied directly to a server, ask what would happen if that machine were rebuilt from scratch tomorrow. Anything that would not come back has to go into the repository or the provisioning, immediately, while you still know why it exists.
The same sweep found two things left behind by an earlier rewrite: a scheduled job still firing every night against code that had been deleted, and a runbook section still telling an incident responder that the data at rest is encrypted in a way it no longer is. Both were reported rather than touched, because they were not that day’s work. Both are the same failure as the privilege file, running in the other direction — the machine remembering something the repository has forgotten.