A developer's local database turned out to be production
A tunnel made the live database answer on localhost, so every migration run that afternoon went to production. Only a typo and a database property saved it.
Can your developer accidentally change your live database? Yes, and the way it happened here is one most teams would miss. A schema migration failed on my machine with a column-name error. I fixed the typo, re-ran it, and only then looked properly at where it had been running.
The connection string in the server’s local environment file pointed at a loopback address on an unusual port. That port was a tunnel, and on the other end of it was production. Every migration, every seed and every development server run that afternoon had been talking to the live database.
The migration I had just run was a schema change against production. It failed because of a typo in one of my own column references. I verified production directly afterwards: migration revision unchanged, table count unchanged, the family and record rows intact. Nothing was lost, and that was luck rather than control.
What was actually going on
A tunnel makes a remote service indistinguishable from a local one, so that every client that talks to localhost can talk to the remote thing unmodified. It works perfectly, and so the hostname in the connection string, the one fact most people use to judge whether they are about to do something dangerous, is what a tunnel makes meaningless. The port was the only tell, and nobody reads a port number carefully.
The other thing that saved it was that the database engine rolls schema changes back as part of a transaction, so the failed migration left nothing behind. I had not chosen the engine for that property and could not have told you that morning which behaviour it had. Several widely used engines apply each schema statement immediately and irreversibly, so a migration that fails on statement seven leaves six applied and no way back except a restore.
What we changed
An address-based check is worthless where tunnels are used. The checks that work ask the database what it is, because that travels with the database and not with the route to it:
- The database name. A destructive script refuses to run unless the connected database’s own name matches an expected pattern.
- A marker row in a settings table saying which environment this is, written at provisioning time. Destructive tooling reads it and refuses on a mismatch.
- An explicit assertion, in the checklist for a later large rebuild, that the connected database is not production, alongside reading the schema from the catalog by hand.
The same kind of mistake appeared in a smaller way. The seed script had a reset flag that truncated every table and re-created the sample data. Run against the shared development database, it destroyed a real account and household that a person had created there a few minutes earlier. That was not production, but somebody had just spent twenty minutes setting up a scenario.
Reset now deletes only the identifiers the seed itself creates. It was proved by creating an unrelated household by hand, re-seeding and confirming it survived. A reset should undo what it did, not empty the room. For a screen-by-screen verification sweep, the test household is now created through the product’s own public endpoints, which exercises the setup path too.
What it did not fix
The near miss happened before any of these checks existed. It was found only because something failed: had my migration been correct, it would have applied cleanly to production and I would have carried on for another hour with no idea. And the engine’s rollback property protects schema changes; it is not a general safeguard against a script that deletes rows.
What to ask your own team or supplier
- How does a destructive script establish which system it is connected to, and is that the address or something the database reports about itself?
- Who can reach production from a developer machine, and through what tunnel or shared credentials?
- Does our database engine roll back a failed schema change, and if not, what is the backup step before each migration?
- Does the seed or reset tooling delete only what it created?
- After a near miss, does anyone check production and write down why it missed?
Where this ends up
It is also why, when a legacy system is replaced while it is still running, the data migration is treated as its own project, with correctness checks on every rehearsal and a way back that somebody has actually used. A migration run once, against a database nobody asked to identify itself, is the version of this story that does not end with a typo.