Editing an order in the portal created a second order
Admins clicked Edit on an order and got a blank form that saved a duplicate. Behind it, any attempt to modify an order crashed the live system.
Topic
Engineering
Architecture, correctness and the bugs that only show up in production.
23 articles
About this topic
Almost everything filed here starts with something that behaved correctly in testing and wrongly in production. That is not an accident of what we choose to write about — it is where the interesting engineering is. A bug that a test catches is a task. A bug that only appears when two requests arrive four hundred milliseconds apart, or when a queue redelivers a message nobody expected to see twice, is a design question wearing a defect as a disguise.
The recurring themes are correctness under concurrency, what a system does at a boundary it does not control, and the difference between code that works and code that can be operated. Where a piece describes one of our own mistakes it says so plainly, including the hypothesis that turned out to be wrong, because the wrong hypothesis is usually the more useful half of the story.
The app said no connection. Our own patience ran out
Users were told to check their wifi. The network was fine. Our own timeouts were shorter than the work, and a generated answer was paid for and thrown away.
The camera button in the field app did nothing
Field staff tapped Camera and nothing happened, so no proof photos. Two separate faults, each enough alone, hidden because the app swallowed the error.
The Start Shift button vanished after one wrong tap
On release day 17 field staff ended their shift by accident and could not start another. The fix reached every phone without a new app release.
The handover photos only showed half the car
Every photo a driver took at a handover reached the office as its top-left corner. Half to nine-tenths of each frame was lost, and none of it can be recovered.
The accounts link said success and lost 87 customers
Customer batches from the accounting package returned success while 87 of 208 records were dropped. Uploads over 100 KB failed. Ten kilos went across as ten grams.
Customers said the confirmation email never arrived
A forwarded Gmail bounce showed every confirmation to a Gmail address refused since one afternoon. Roughly 37 bookings went unconfirmed, and no code was at fault.
The portal went down for a few seconds, many times a day
Staff said the server was down. It was healthy and answering in 40 milliseconds, yet the office saw 219 error pages in a day. The cause was a customer's browser.
A release gate that refuses what a CMS lets through
In a CMS a page with no description or an empty FAQ publishes. With a typed schema the build fails instead. We hit it four characters over a limit this week.
Reps' phones showed the wrong currency and dates
A sales app hardcoded currency, language and timezone in about 120 places because a rep's phone was not allowed to read the company's own settings.
A failing test run that said nothing about the code
Three test suites failed on code nobody had changed. Leaked test users had filled a shared tenant to its seat limit, and a red run meant nothing about the release.
Moved your pricing to new software? 102 rules went missing
A fabricator's pricing moved from a spreadsheet into new software. The import said it worked, yet 102 rules were missing. How we found them and why it matters.
The mobile app worked for weeks, then would not build
The first release build of the mobile app failed on two configuration bugs weeks old. Development mode had never taken either code path.
Invoicing software charging tax on zero-rated sales
A sale that should carry no tax could be invoiced at 18%, because a zero rate was mistaken for no rate. Nothing warned anyone.
Seventeen passing layout tests that could never fail
Seventeen layout tests passed against a stylesheet that made the assertion impossible to fail. Fifty-one screen tests passed on a portal that was read-only.
Who moved fifty accounts? Only a good audit log can tell you
Someone reassigned fifty dealer accounts and now denies it. Whether your software can name which accounts and who did it depends on how the log was built.
An approve button that never approved anything for months
An expense system had an approval step no expense could reach. Before rebuilding it, we took out everything that was not doing work. Three tests for what is real.
Secure a brochure site by not running an application on it
Two compromised sites were rebuilt as static HTML with no interpreter and no database. The attack surface did not shrink — it stopped existing.
A receivables ledger seeded once and never true again
Two tables held outstanding amounts. Nothing kept them in step with the invoices they came from. Pay an invoice and the ledger never noticed.
They built the client's deck by hand for every single pitch
A proposal is a slide deck and a cost sheet. Generating both from the same records that produce the booking removes an afternoon of retyping per pitch.
The booking screen named a driver who was never assigned
In a car rental system, nine places answered who the driver was. Eighty-one legs got an arbitrary answer, sixteen named the wrong person, seventeen a phantom.
Staff who can see invoices should not mark them paid
Two hard-coded roles made everyone an administrator. How 77 permissions replaced them, and the bug that nearly showed field staff the director's screen.
Booking site down for hours with hardly any visitors
A booking site went down for four hours with about twenty requests a minute and an idle database. A bigger server would have made it worse.