A green test nobody has seen fail is not evidence

A responsive-layout test suite asserted that the page’s scroll width never exceeded the viewport width. Seventeen screens, seventeen passes, across every breakpoint.

The stylesheet set horizontal overflow to hidden on the root elements — a common backstop to stop stray content producing a horizontal scrollbar. That declaration suppresses the page scrollbar, which means the measured scroll width can never exceed the viewport width, which means the assertion could never fail.

Seventeen green tests proving nothing. Not a subtle sampling problem — the assertion was mathematically incapable of failing on any page in that application.

The check that makes a test real

The rewrite measured actual geometry: find any element whose right edge falls beyond the viewport with no scrolling ancestor to contain it, and name it.

Then — and this is the step that matters — the new test was proved non-blind by injecting an oversized element into a narrow page to confirm it fired and correctly identified the offender.

That is the whole discipline, and it can be stated in one line: before trusting a test, watch it fail for the reason you think it detects.

A passing test tells you two things are consistent: the assertion and the current behaviour. It tells you nothing about whether the assertion is capable of distinguishing correct behaviour from incorrect. Those are different properties and only one of them is what you wanted.

The immediate follow-up in the same work is a nice illustration of how easy this is to get wrong twice. A test asking “is this table stacked on mobile?” checked whether the header height was at most one pixel. A correctly stacked header measures two pixels. The test reported a correct screen as broken. Same category of error, opposite direction, in the replacement for the first one.

Four more from the same family

Fifty-one screen tests passed against a read-only portal. Every screen rendered, every assertion about content held. None of them exercised a write. The note in the log is exact: rendering is not working. A test suite that only ever asks “does this draw” will be entirely green on a product where nothing can be saved.

A deploy verification that could not fail. The deploy script curled the application immediately after restarting the process manager, inside the restart window, and reported the resulting empty response. It did that on every healthy deploy. It was noted roughly fifteen times over seven months before anyone fixed it, because a verification step that always reports the same thing becomes background noise within a week. A second deploy script had the same defect and additionally compared against a fallback page rather than the application, reporting a six-kilobyte placeholder as though it were the three-hundred-kilobyte bundle.

A build reported as successful when the success belonged to something else. A build was piped through another command, and the exit code that was checked belonged to the last stage of the pipeline rather than the compiler. The pipeline succeeded at its job of printing lines. Nobody asked the compiler.

A page that could not be probed. The application is a single-page app, so the server returns a successful response for any path at all. Checking whether a particular page had been deployed by requesting its URL proved nothing. It went unnoticed until an external reviewer rejected the listing for a missing page. Deploy verification now greps the served bundle for text belonging to that specific page.

The common structure

In each case the check was present, ran routinely, and was incapable of producing the negative result it existed to detect. The reasons vary — a stylesheet suppressing the signal, a timing window, an exit code from the wrong process, a server that answers everything — but the shape is identical.

I have come to think of these as inert controls, and they are worse than absent ones, because an absent check is a known gap and an inert check is a false assurance that stops anyone looking.

The same pattern shows up outside testing. A security plugin writing access rules to a file the web server never reads. A certificate renewal cron entry that self-disables. A noise-reduction filter whose strength parameter did nothing, removing an identical amount at every setting. A silence threshold carried over from a different recording environment that sat below the noise floor and so never matched anything.

That last pair are instructive because they were caught the same way: by checking that changing the input changed the output. If turning a control up and turning it down produce the same result, the control is not connected to anything.

Practical habits

Prove the test fails. Break the thing deliberately, watch the test go red, fix it, watch it go green. On a bug fix, stash the fix and run the new test against the unfixed code — a security fix in one of these systems was verified this way and produced fourteen failures against the original code, three of them the specific leak being fixed. Without that step you cannot tell a regression test from a tautology.

Distrust checks that have never been red. A test that has passed since the day it was written, across many changes, is either covering something genuinely stable or covering nothing. It is worth knowing which.

Verify the artefact, not the configuration. Two mobile permissions shipped because a runbook’s description of the manifest was trusted rather than the built manifest being read. A dashboard showed a fingerprint that produced no corresponding entry in the generated file. In both cases the only trustworthy check was reading what was actually produced.

Beware the phrase list. A cleanup was verified by searching for six known strings and declared complete. A seventh, worded differently, shipped. The remedy was applied to the verification rather than the code: match claim-shaped patterns rather than exact strings, and confirm the new check fires on the exact string that got through. A phrase list is not a check — it is a check for the instances you already thought of.

Treat a completion report as a claim. Where work is parallelised across people or tools, a report saying all tests pass is a claim, not a fact. In one case the real number was three suites short of what was reported; in another the tests were green but stale under a contract change. Re-run the gates yourself. This is not distrust of the person; it is recognition that “I ran it and it passed” is a statement about a moment that has since gone.

The principle

A test is a measuring instrument, and an instrument you have never seen respond to a known input is not calibrated.

The question to ask of any check in your system — test, monitor, alert, verification step, security control — is simply: when did this last tell me something I did not already believe? If it never has, find out whether it can.