The one convenience that turned a single breach into an estate-wide one

Somewhere in the configuration of every site on the server was the same variable. One value, reused as the database password for every site, the administrative account on several of them, and the file transfer account. It had a name that made its purpose obvious. It had been convenient for years.

When one site was compromised and the attacker could read that site’s configuration file, they did not obtain one database password. They obtained the password to every database on the machine, and the credentials to log in as an administrator on multiple unrelated sites.

The password itself was not weak. It was never brute-forced and never guessed. It was read, once, out of a file that the application must be able to read in order to function at all. Complexity requirements are entirely irrelevant to that failure mode.

What a credential is actually for

Access control is usually discussed in terms of strength — how hard is this to guess. That framing is only half of it, and on a compromised host it is the less important half.

The other half is scope. A credential answers two questions, not one:

1. How hard is it to obtain without authorisation?
2. Once obtained, what does it open?

Strength addresses the first. Nothing except separation addresses the second. And on a server where the application must read its own database password out of a configuration file, the first question is largely settled the moment code executes in that application’s context. There is no cryptography protecting a plaintext value that the process is required to be able to read.

So the design assumption should be: any credential stored on a host will be readable by anything that achieves code execution on that host. Not “might be”. Will be. Once you accept that, the only control still available to you is what the credential opens. A shared password is a decision to make that answer “everything”, and it is a decision made once and then forgotten about for years.

The convenience is the trap

Nobody chooses a shared password because they think it is safe. It gets chosen because it makes a dozen ordinary tasks easier, and each of those tasks is real:

• Onboarding a new site is one fewer value to generate, record and store.
• Debugging is faster because you already know the credential for whatever you need to inspect.
• Bulk operations across sites need no lookup table.
• Nobody gets locked out because a value was recorded incorrectly somewhere.

The saving is small, daily, and visible. The cost is enormous, rare, and invisible until the day it lands entirely at once. That asymmetry is exactly the shape of decision that organisations make badly by default, because the feedback loop only closes on the worst day.

The same asymmetry produces the surrounding habits. The shared value ends up committed to a repository, because it was easier than a secret store. It ends up in a document, because someone needed to share it. It stays valid for years, because rotating it means touching every site simultaneously and therefore risking every site simultaneously — and so a shared credential is structurally difficult to rotate. Its reuse is precisely what makes it permanent.

That is the part worth dwelling on. Reuse does not only widen the blast radius. It disables the remedy. A per-site credential can be rotated on a Tuesday afternoon with a single site’s downtime as the worst case. A shared one requires a coordinated change across the estate, so it never happens, so the value in that document from three years ago is still live.

What we did

Every database password on the machine is now distinct per site. The shared variable is gone from the environment files entirely, and the old name is documented as dead so nobody reintroduces the pattern out of habit. Administrative accounts are per site. The file transfer account no longer shares a value with anything.

Remote shell access moved to key-only. Password authentication was switched off completely, not merely discouraged, which means a stolen or reused password is not a login vector at that layer at all. The console password still exists for recovery through the hosting provider, and the documentation is explicit that it is only that — because the failure mode we were guarding against is somebody finding a still-valid-looking credential in an old file and assuming it works everywhere.

That last point has a cost we accepted deliberately: with a single trusted key and no password fallback, losing the key means recovery through the provider’s rescue console. The right answer is a second trusted key held separately, and the documentation says so plainly rather than pretending the single-key setup is finished.

Rotation in an environment you did not build

If you have inherited an estate and suspect this pattern, the ordering matters, because doing it in the wrong sequence locks you out or leaves the attacker in.

First, establish an access path that does not depend on any credential that might be compromised. Keys, out-of-band, verified working, before touching anything else. Rotating credentials while your only route in uses one of them is how a hardening exercise becomes an outage.

Second, enumerate every place the shared value appears. This is the step people underestimate. It will be in application configuration files, deployment scripts, environment files, backup jobs, monitoring checks, scheduled tasks, a spreadsheet, and at least one document nobody remembers writing. If you rotate before you enumerate, you find the remaining locations by watching things break.

Third, rotate per system rather than all at once, so a mistake affects one site. This is only possible because you are moving to distinct values — which is a small illustration of why the separated design is easier to operate as well as safer.

Fourth, and most importantly on a host that has been compromised: rotation after a breach is not cleanup, it is a race. If the attacker still has code execution, they will read the new values out of the configuration files the moment you write them. Credentials must be rotated after the persistence mechanisms are removed and verified gone, not before, or you have simply told the attacker your new password.

The principle

Treat every stored credential as already disclosed, and design the boundary around what it opens rather than how hard it is to guess.

Two questions to run against any environment you are responsible for:

• Pick any single credential in the system. What is the complete list of things it opens? If the list has more than one entry, you have a shared blast radius, whether or not you think of it as one.
• When was it last rotated, and could you rotate it this afternoon? If the honest answer is “not without coordinating across everything”, it is shared in practice regardless of what the configuration looks like.

Distinct credentials per system cost a few minutes each at setup. Shared ones cost you the whole estate, once, on a day you do not get to pick.