The local database was production, and a typo of mine is the only reason it survived

A schema migration failed on my machine with a column-name error. I fixed the typo and re-ran it, and only then did I look properly at where it had been running.

The connection string in the server’s local environment file pointed at a loopback address on an unusual port. That port was a tunnel. On the other end of it was production.

Every migration, every seed, every development server run that afternoon had been talking to the live database.

The migration I had just run was a schema change. It failed because of a typo I had made in one of my own column references, and the engine’s transactional DDL rolled the entire thing back as a unit, leaving nothing behind. I verified production directly afterwards: migration revision unchanged, table count unchanged, all the family and record rows intact.

So the sequence is: I pointed at production without knowing, ran a schema change against it, and was saved by a spelling mistake and by a property of the database engine I had not thought about that morning.

Why a tunnel is exactly the wrong shape

The instinct after something like this is to add a check that the host is localhost. That check already passes. The host is localhost. That is what a tunnel does.

A tunnel exists to make a remote service indistinguishable from a local one, so that every client that knows how to talk to localhost can talk to the remote thing without being modified. It is a good tool and it works perfectly. The consequence is that the single piece of information most people use to judge “am I about to do something dangerous” — the hostname in the connection string — is the one piece of information a tunnel is designed to make meaningless.

The port was the only tell, and a port number is not a fact anybody reads carefully.

Any safety check based on the address of a database is worthless in an environment where tunnels are used. The check has to be based on something that travels with the database rather than with the route to it.

The ones that actually work:

• The database name. A destructive script refuses to run unless the connected database’s own name matches an expected pattern. The name comes back from the server, so a tunnel does not disguise it.
• A marker row. A single row in a settings table saying which environment this is, written at provisioning time. Destructive tooling reads it and refuses on a mismatch.
• An explicit negative assertion. Before a large rebuild later in the project, the checklist included asserting in so many words that the connected database is not production, alongside reading the schema out of the catalog by hand rather than trusting the application’s model definitions to describe it accurately.

All three are cheap. What they have in common is that they ask the server what it is, rather than asking the client where it thinks it is pointing.

Transactional DDL is not a fact about your code

The other thing that saved this was that the database engine rolls schema changes back as part of a transaction. A migration that fails halfway leaves nothing.

Not every engine does that. Several widely used ones apply each schema statement immediately and irreversibly, which means a migration that fails on statement seven leaves six applied and no way back except a restore.

I had not chosen this engine for that property and could not have told you, that morning, which behaviour it had. It is worth knowing before you need it, because it decides what a failed migration costs. If your engine does not have it, the cost of a failed migration is a restore, and the process has to reflect that — one statement per migration, a verified backup immediately before, and a tested down path.

The reset that deleted a real person’s work

The same class of mistake appeared again in a much smaller way, and it is the one that changed my habits most.

The project’s seed script had a reset flag. Reset truncated every table in the schema and re-created the sample data.

Run against the shared development database, it destroyed a real account and the household that a real person had created there a few minutes earlier. Not production. A development database, which is exactly the sort of place where people assume the data does not matter, and where somebody had just spent twenty minutes setting up a scenario.

Reset now deletes only the identifiers the seed itself creates. Nothing else is touched. That was proven the only way worth proving it: create an unrelated household by hand, re-seed, confirm it is still there.

A reset should undo what it did, not empty the room. Truncating every table is not a stronger version of cleaning up after yourself; it is a different operation that happens to include it. The distinction only becomes visible when something you did not create is present, which in a shared environment is always.

A related habit came out of the same period. For a screen-by-screen verification sweep, the test household was created by calling the product’s own public endpoints rather than by running the seed script against the database. That is slower. It also means the setup path is itself exercised, and nothing that runs is capable of writing to a table the API would not write to.

The rule

If a tool can destroy data, it must establish which system it is connected to by asking that system, and it must refuse rather than proceed when the answer is not what it expected. Address-based checks are not that, because addresses are routable, and the whole discipline of operations is built on making one address stand in for another.

Take the smaller lesson too. The reason I found this at all is that something failed. Had my migration been correct it would have applied cleanly to production, and I would have carried on for another hour with no idea. A near miss is more useful than an incident, but only if you go and look at why it missed.