Editing an order created a duplicate order: why
Edit opened a blank form that saved a second order, and changing a completed order risked wiping shipped quantities. How to check your system.
Topic
Engineering
Architecture, correctness and the bugs that only show up in production.
23 articles
About this topic
Almost everything filed here starts with something that behaved correctly in testing and wrongly in production. That is not an accident of what we choose to write about — it is where the interesting engineering is. A bug that a test catches is a task. A bug that only appears when two requests arrive four hundred milliseconds apart, or when a queue redelivers a message nobody expected to see twice, is a design question wearing a defect as a disguise.
The recurring themes are correctness under concurrency, what a system does at a boundary it does not control, and the difference between code that works and code that can be operated. Where a piece describes one of our own mistakes it says so plainly, including the hypothesis that turned out to be wrong, because the wrong hypothesis is usually the more useful half of the story.
App says no internet connection? Check the request timeout
An app told users to check their wifi when the network was fine: our own timeouts were shorter than the work. How to tell, and what we changed.
Camera not working in your field app? Check permissions
A field app's camera button did nothing because two permission faults hid behind swallowed errors. What to check on your own app, and what we fixed.
Clock in app: Start Shift button gone after one tap
On release day 17 staff ended their shift by accident and could not start another. The fix reached every phone without a new app release.
Vehicle inspection app photos arriving cropped: the cause
Handover photos from a rental inspection app arrived as a corner of the car. Large phone photos were being cut, not shrunk. How to check yours.
Tally integration says success but customers are missing
A Tally link reported success while 87 of 208 customers were dropped and 10 KG arrived as 10 GM. What to check on your accounting software integration.
Emails not delivered to Gmail? Why confirmations bounced
Every booking confirmation to a Gmail address was refused from one afternoon. About 37 went unconfirmed and no code had changed. How to find the cause.
Portal down for seconds, several times a day: the cause
Staff saw error pages in bursts while the server looked healthy. A customer's browser pass was crashing the portal, which restarted 340 times.
How to stop pages going live without a description
A CMS lets a page with no meta description publish. A typed content schema fails the build instead. How that works, and what it costs your editors.
Wrong currency, language or dates in your sales app?
Reps see the wrong currency or dates when an app has them written into its code. Why that happens, how to check yours, and what we changed.
Flaky tests failing when nobody changed the code
Tests failing when nobody changed the code usually means a shared test environment filled up. How we found it, and how to check yours.
Data migration testing: 102 pricing rules went missing
How to check that a spreadsheet-to-software move lost nothing. Ours imported clean and was still 102 rules short. The count-by-count test that found them.
App worked in debug but the release build failed
A mobile app worked for weeks in development and failed on its first release build. Development mode never ran the code paths that broke. What to check.
Zero-rated supply invoiced with tax: how it happens
Invoicing software can charge 18% on a zero-rated sale because it reads a zero rate as no rate. How to test yours and what to ask your supplier.
False positive tests: how tests pass on broken features
Tests can pass while the feature is broken, because the check was never able to fail. Examples from our own work, and how to test your supplier's tests.
Who changed fifty accounts? What an audit trail must hold
When someone denies changing fifty accounts, only an audit trail that names the records and the person can settle it. What to check in yours.
Expense approval workflow not working? Fix before rebuild
An approval step no expense could reach, because every claim was approved on creation. Three tests for what to remove before you decide to rebuild.
Is a static website more secure than WordPress?
A static site has no database, login or plugins to attack. Two compromised brochure sites were rebuilt that way. When it is the right answer, and when not.
Receivables ledger not matching invoices? Why it drifts
A receivables ledger kept in its own table drifts from the invoices after every payment. How to check yours, and why we worked the figure out instead.
Proposal automation: generate the client deck from bookings
Building every client pitch deck by hand means retyping what the booking already holds. How to check yours, and how we generated the deck and cost sheet.
Car rental software showing a different driver per screen
Nine screens worked out the driver in nine ways: 81 legs got an arbitrary answer, 16 the wrong name, 17 a phantom. What we changed and how to check yours.
User roles and permissions: staff see only what they should
Two hard-coded roles make everyone an administrator. Why roles should be built from fine-grained permissions, and how 77 permissions replaced them.
Website down with hardly any visitors: not server overload
A booking site went down for four hours with hardly any visitors and an idle database. Why a bigger server would not have helped, and how to tell busy from blocked.