Store everything in UTC, but never decide anything in it

A smoke test reported that a discount scheme which expired yesterday was still being applied to orders. That went into the defect register as a money-path bug: expired promotions keep discounting. It is exactly the kind of finding you act on immediately.

It was not a bug. The test computed “yesterday” from the local date on the machine running it. The server evaluated the scheme’s validity against the UTC date. The local timezone is five and a half hours ahead of UTC, so between midnight and half past five in the morning local time, local-yesterday is still UTC-today. The scheme was genuinely live. The test was asking a different question from the one it thought it was asking, and only during a five-and-a-half-hour window each day.

So the reported defect was false. Underneath it was a real one, smaller and more consequential.

The actual bug

A discount scheme’s validity period is a business date. When a business says a promotion runs until the 31st, they mean until the end of the 31st where the business is. They do not mean until 23:59:59 UTC, and if you are five and a half hours ahead of UTC that distinction costs the last five and a half hours of the promotion.

The system was evaluating it in UTC. There was a tenant timezone setting — stored, editable, shown in the administration screens — and it was read by nothing outside its own CRUD. A configuration field that exists and influences no behaviour is worse than no field at all, because it tells everyone the problem has been handled.

The same class of error turned up twice more in the same codebase. An attendance smoke suite failed eight checks once at a date rollover and passed on re-run. And the trial-expiry gate compared against UTC, which meant a customer in that timezone lost the last five and a half hours of their trial period — a small injustice, applied automatically, to every single customer.

The fix was a pair of helpers that resolve the current business date and timezone for the tenant, reading the tenant identity from the request context rather than from a parameter anyone could forget to pass, consumed by the scheme evaluation and everywhere else that asks “is today within this range”.

The distinction that prevents all of this

There are two different things that both get called “a date”, and conflating them is the root of every bug above.

An instant is a point on the universal timeline. When a row was written. When a payment cleared. When a user logged in. These have one correct representation, it is UTC, and they should be stored and compared as UTC without exception.

A business date is a label a human applies to a period of activity in a particular place. The day a promotion ends. The day a trial expires. Which day’s sales figures a transaction belongs to. Which day an attendance record counts against. These are not instants. Converting them to instants requires a timezone, and the correct timezone is a property of the business, not of the server, not of the database, and not of the machine running the test.

Almost every timezone bug I have seen is a business date being handled as if it were an instant. The conventional advice — “always use UTC” — is correct for instants and actively misleading for business dates, because it sounds like a complete rule and it addresses half the problem.

The more useful formulation: store instants in UTC, and resolve business dates through an explicit timezone that belongs to the entity the decision is about. In a multi-tenant system that is the tenant’s timezone. In a single-tenant system it is still worth making explicit, because “the server’s timezone” is a value that changes when you move hosting provider.

What made it hard to see

Three properties combine to make this class of bug unusually durable.

It is correct most of the day. A five-and-a-half-hour offset means the wrong answer appears in roughly a fifth of the day. Any test that runs in the other four-fifths passes. Re-running a failing test at a different hour makes it pass, which is the single most effective way to convince a team that a failure was flaky and can be ignored.

The magnitude is small enough to be invisible. Losing the last few hours of a trial does not generate a support ticket. Nobody notices that a promotion ended slightly early. The bug does real damage in aggregate and none of it is individually reportable.

The test and the code can be wrong in different directions. Here the test’s error and the code’s error were both timezone-related but not the same error, so fixing one did not reveal the other. The test was fixed by giving it a seven-day margin — so that a one-day skew can never masquerade as expiry again — and the code was fixed separately by introducing the business-date helpers. Two changes, two root causes, one symptom.

That last point generalises. When a test fails intermittently around a boundary, suspect both sides. The instinct is to decide whether the test or the code is wrong. Boundary bugs are very often both, because they come from the same underlying confusion expressed in two places.

Practical rules

A few things I now do by default:

Give tests a margin. A test asserting that something expired yesterday is testing a one-day boundary with unknown timezone assumptions on both sides. Test with seven days. If seven days ago does not count as expired, you have a real bug rather than an offset.

Make the timezone an argument, not an ambient value. Anything that resolves a business date should take the timezone explicitly, sourced from the entity the decision concerns. Code that reads the process timezone is code that behaves differently on a developer’s laptop and in production, and that difference is invisible until it is expensive.

Audit configuration fields for whether anything reads them. The tenant timezone was stored and unused for a long time. It is a cheap and revealing check: for each configuration field, grep for reads outside its own CRUD. Any field with none is either dead or a bug waiting to be noticed, and you cannot tell which without looking.

Write down which of your date fields are instants and which are business dates. Not in a document nobody reads — in the column comments, or the type names, or both. The bug is not that people handle business dates wrong; it is that they cannot tell which kind they are looking at from the field name. expires_at and valid_until look like the same kind of thing and are not.

The principle

Timezone handling fails at the moment a decision is made, not at the moment data is stored. Storage is a solved problem and everyone has solved it. Comparison is where it goes wrong, because comparison requires answering “what day is it”, and that question has no correct answer until you say who is asking.