An enforcement rule with no operator surface is an outage waiting for a date

A routine reconciliation ran every check in the project rather than trusting the tracker. Twenty-five of the thirty-five API test suites failed. No release had gone out. Nothing had been merged. The last code change to any of the affected areas was weeks old.

The cause was not a regression. The shared development tenant’s trial period had ended four days earlier, and the trial-lifecycle enforcement shipped a few weeks before was doing precisely what it was built to do: refusing every write.

It was proved rather than assumed, and the proof is the part worth copying. Every failing suite failed at its first write, with the trial-expired error code. And the ten suites that passed were exactly the ten that register a fresh tenant of their own rather than using the shared one. That partition is not a coincidence you could talk yourself into — it is a signature, and it identified the cause in minutes rather than sending anybody looking for a broken commit.

When a large number of tests fail at once, look for the partition between the failures and the passes before looking at any individual failure. If the two groups differ by something other than what they test, the cause is environmental.

The environment had been read-only for four days

That is the operational part. Testers had been blocked. The browser test suite could not run, because it cannot sign in and create anything. And nobody had reported it, which means the four days were spent either working around it or not working.

Nothing had warned. No banner, no email, no log line anybody was watching. The gate simply began returning refusals on a Tuesday.

The finding is a product finding

The engineering conclusion here would be to extend the shared tenant’s trial and move on. That is a five-minute fix and it is the wrong conclusion.

The trial gate is correct. It expires trials, it refuses writes, and it does so without a scheduled job — expiry is computed on read, which is the right design because a cron that fails leaves customers on a permanent free trial. There is nothing to fix in the enforcement.

What is missing is everything around it. Nothing warns a tenant that the trial ends in three days. Nothing warns them on the day. Reactivating a tenant is an API call on the platform-administration plane with no interface at all, which means the only people who can unblock a customer are the people who can issue authenticated requests by hand.

And the same sequence will happen to the first paying customer whose card fails. They will find out when their staff cannot save an order.

That moved the administrative console from a polish item, scheduled somewhere after the features, to an operations requirement that has to exist before the first customer does. Enforcement without an operator surface is not half a feature. It is a scheduled outage with the date already set.

The gate itself had been wrong twice

Getting to a correct gate took an audit round, and both of the earlier errors are worth naming because they are easy to repeat.

It compared against the wrong calendar. Expiry was evaluated against the universal date rather than the tenant’s business date, so a tenant several hours ahead lost the last few hours of its trial. Small, automatic, and applied to every customer. The same confusion had already been found and fixed once in the discount engine, which is what a class of bug looks like when only one instance gets fixed.

It refused everything, including reads. An expired trial locked the customer out of their own data completely. That is the wrong posture on any reading of the business situation: a customer whose trial ran out is somebody you want to convert, and the first thing they experience is being locked out of records they entered themselves. It is also useless as a prompt, because the screen that would tell them how to subscribe is behind the same gate.

The corrected posture separates three states. An expired trial or a suspended tenant blocks writes and preserves reads — you can see everything, you can change nothing, and the interface explains why. A closed tenant blocks everything, and closed had not been checked at all before.

The implementation detail that makes it survive: the rule is expressed in terms of the HTTP method rather than enumerated per route. A new endpoint added next year is covered because it is a write, not because somebody remembered to add it to a list. Any rule that requires enumeration will eventually be missing an entry, and it will be the entry that matters.

Two things that only look obvious afterwards

Reactivation must be exempt from the gate it lifts. The endpoint that reopens a closed tenant cannot itself be blocked for closed tenants, or closed becomes a state with no way out. This is obvious when stated and it is exactly the kind of thing found by a test rather than by reasoning, because the reasoning tends to stop once the gate works.

Enable enforcement only after measuring the population it will affect. Before the lifecycle gate went live on the shared environment, the tenant table was queried to confirm how many of the 126 tenants had an already-expired trial. The answer was zero, so switching it on changed nothing that day. Had it been forty, switching it on would have locked out forty tenants at once, and the first anyone would have known is the support queue.

That check costs one query and it is the difference between a deployment and an incident. It applies to every rule that begins refusing things: rate limits, quota enforcement, validation tightening, mandatory fields. Count who fails the new rule before you turn it on.

Rules

Look at the partition first when many tests fail together. Passes and failures divided by something other than subject matter means the environment, not the code.

Never ship an enforcement rule without the surface that resolves it. A warning before, a clear message during, and a way for an operator to lift it without hand-crafting a request.

Block writes, preserve reads, for any commercial state you want the customer to recover from.

Express lifecycle rules by request method, not by route list. Lists get out of date one route at a time.

Exempt the recovery path from the gate it recovers from.

Count the affected population before enabling any rule that refuses things.

The part I got wrong was assuming the gate was finished when the enforcement passed its tests. The enforcement was the easy half. The half that costs money is what happens to the person on the other side of the refusal, and that half had no owner until an environment went read-only for four days and nobody noticed.