I reported a build as passing, and it was the exit code of the command I piped it into
A mobile release build produces a great deal of output, so I piped it through a command that shows only the last part. The command succeeded. I read that as the build succeeding and said so.
In a shell pipeline the exit status is the status of the last command. The pager exits zero whatever it was given. The build had failed, the failure was several hundred lines above what I was looking at, and I had reported a success that belonged to a completely different process.
I corrected it in the same session, which is the only reason it did not become a fact somebody else planned around. It is a small mistake with a general shape: the thing you measured and the thing you reported were not the same thing, and nothing in between complained.
Reports are claims
That was mine. The same class kept appearing from every direction.
One round reported all twenty-five test suites green. Re-running them produced twenty-two green and three failures. The report was not dishonest — it described a run that had happened, at a moment when it had been true, or at least when its author believed it. It was simply not a verification, because nobody had checked it against the tree as it stood.
Another round left two lint errors in the codebase and finished without producing a report at all. Absence of a report reads as absence of a problem, which it is not, and the errors were found by running the gate rather than by reading anything.
The habit that came out of this is simple and slightly tedious: run the gates yourself, at the end, against the tree you are about to hand over. Lint, type check, migrations forward and back, the test suites, the browser suite. Not because anybody lied, but because a report is a statement about a past state of a working copy, and a gate is a statement about the current one.
The value is not in catching people out. It is that the report and the gate disagree often enough to be worth the twenty minutes, and every disagreement is a thing that would otherwise have been discovered by somebody downstream at a worse moment.
The conditions under which a result means anything
There is a second-order problem that took longer to see.
Several people, or several processes, working in parallel against one shared development database means a full test run measures the database as much as the code. A batch run showed two failures that passed when the same suites were run individually. The cause was not the code. Another process was writing to the same tenant at the same time.
So a full-suite result acquired a precondition: it is only authoritative when nothing else is touching the environment. Runs made while other work is in flight are useful for spotting obvious breakage and worthless as evidence. That distinction now gets stated when a result is recorded — an uncontended run is a claim about the code, a contended one is a claim about the afternoon.
This applies well beyond parallel agents. It is the same reason a flaky integration suite on shared infrastructure erodes trust in itself: not because the tests are bad, but because nobody records the conditions, so every result is equally believable and equally worthless.
A test result without its conditions is not evidence. Record whether the environment was exclusive. If it was not, say so, and do not use the number.
What a good report contains
Having been on both sides of this, the reports that turned out to be worth reading had the same properties.
The command, and its real output. Not “tests pass” but the suite names and the counts. A count that changes between rounds is informative; the word “pass” is not.
What was not verified. Several rounds of mobile work ended with an explicit statement that no screen had been driven on a device — type checked, contract checked, endpoints exercised against the live API, but never run by a human on hardware. That sentence is more useful than any of the green ticks above it, because it tells the next person exactly where the risk sits.
Deviations from what was asked. Where a module was built differently from its brief, or a file outside the stated scope was touched, saying so in the report is what keeps the brief meaningful. Silent deviations are how a scope becomes decorative.
Adjacent problems found and not fixed. Every round turned up defects outside its own remit. The right move is neither to chase them nor to ignore them, but to log them precisely enough that somebody can pick them up. A report that only describes what went well is a report that has thrown away half of what the work produced.
The uncomfortable part
The reason this matters more than it used to is that a lot of work is now done by processes that report on themselves. A summary saying everything passed is generated with exactly the same confidence whether or not it did, and it is fluent either way. The fluency is the problem — a badly-written report from a tired person carries useful signal in its raggedness, and a generated one does not.
The only defence I have found is boring. Run the thing. Read the actual output. Check that the number in the report is the number the tool produced. And when a result depends on conditions, write the conditions next to it.
I do not exempt myself from this, and the reason I do not is at the top of this article.
Rules
Check what your exit status is actually measuring. A pipeline reports its last command. So does a wrapper script, and so does a helpful test runner that swallows a non-zero return.
Re-run the gates against the tree you are handing over. The report describes a working copy that no longer exists.
Record the conditions of a test run. Shared environment or exclusive. Without that, the number is not evidence.
Say what you did not verify, explicitly. It is the most useful line in any report.
Log adjacent defects rather than chasing or ignoring them.
A report is a claim about work. A gate is a measurement of it. They are different objects and the habit of treating them as the same one is how a project quietly develops a version of itself that only exists in its own status updates.