The security fix took the quoting system down. Nobody knew

Your quotation system is where every job starts. A customer asks what a kitchen line will cost, somebody in the office opens the system, builds the quotation and sends it. On a day the system is down, that person opens it and gets an error. If nobody is quoting that day, nobody notices. If somebody is, the call comes to you.

Now imagine that what broke it was a security fix. A hardening pass had gone across the whole server the system lives on, tightening who may reach what. It was the right thing to do. One of the rules it removed was the one this system was quietly relying on.

The system belongs to a commercial kitchen fabricator, JK Kitchen, and they did not report it. We found it on 3 September while starting unrelated work: the check that should answer “healthy” answered with a server error, and every attempt the system made to reach its own database was refused. The rule it depended on had been removed on 19 August.

We are not going to tell you how long it was down, because the log does not record it. It records the day the rule was removed and the day the fault was found, and nothing in between. Whether the first failure came that evening or weeks later is not something we measured, and a number we did not measure does not belong in an article. What we can say is that in that window nothing raised an alarm, and the thing that found it was a person opening the system for another reason.

What was actually going on

The quotation system and its database live on the same machine. The system had been told to reach the database by the machine’s public address, the one the outside world uses, rather than the private address that means “this same machine”. The database keeps a list of who may connect from where, and the hardening pass went through that list removing anything that looked like access from outside, because access from outside is exactly what a hardening pass exists to remove.

From the database’s point of view, from that day on, the quotation system was a stranger knocking from the internet, and it was correctly turned away. Neither decision was wrong on its own. The system’s address was a shortcut that had always worked; the security change was a good one. The fault was that nothing connected the two, so a rule could be removed without anyone knowing what depended on it.

There was a second thing, found in the same hour. The documented super-administrator login did not work at all. The password written in the system’s configuration and the one the database was actually checking against had drifted apart at some point, one changed and the other not, and the login that the documentation promised had been dead for as long as that gap existed.

What we changed

We pointed the system at its own machine by the loopback address instead of the public one. A database backup taken minutes earlier over that same address had already proved it worked, which is why it was the safe choice rather than a guess. Restarted the service, confirmed the health check answered correctly and that a login went through. Then set the super-administrator password in the database to match the one the configuration declared, so the documented login did what the documentation said.

What it did not fix

Nothing was added that day to catch it next time. The log records the fix and does not record an alert, so a system that fails the same way tomorrow would be found the same way: by whoever happens to open it. A health check that answers correctly is only useful if something is asking it, and nothing was.

The hardening was right. The dependency on the public address was the fault, and it was a fault before the hardening as well; the hardening only made it visible. Any other system on that machine with the same shortcut in its configuration would have failed the same way, and the fix here was to this one.

And the password mismatch was corrected, not the habit that let it drift. Two places held the same secret and nothing checked they agreed.

The mechanism, in plain words

PostgreSQL decides who may connect using a file of host-based rules: this user, from this address, to this database, by this method. The application’s connection string named the server’s public IP, so it matched a rule permitting that address. The hardening removed that rule, and from then on every new connection was rejected with “no pg_hba.conf entry”, the application’s health endpoint returned 500, and there was nothing watching the endpoint. Repointing the connection string at 127.0.0.1 matched the loopback rule that hardening never touches.

The transferable part is short. An application on the same machine as its database should reach it over the loopback, never over the public address. A security change that removes access rules should be followed by a walk through every service that has to reconnect, because “nothing complained” only means nothing reconnected yet. And a health check nobody polls is a light in an empty room.

Where this ends up

Sazinga Quote grew out of this fabricator’s system, and the fault above is the kind that a quotation product has to be watched for, because the day it fails is the day a customer is waiting for a price.