Build passed but actually failed: the pipefail trap

A build can be reported as passing when it failed because, in a shell pipeline, the exit status belongs to the last command and not to the build. If the build output was passed through a command that only shows the end of it, that command succeeds whatever happened above. In bash the setting that makes the pipeline report the failure is called pipefail. The business question is simpler: which command produced the pass in your supplier’s report, and was it the build itself?

Here is how it happened to us, in a report I wrote.

How do you know your supplier’s report that all tests pass is true? In this case, the honest answer is that I reported a failed build as passing, and it was my own report.

A mobile release build produces a great deal of output, so I piped it through a command that shows only the last part. That command succeeded, and I read it as the build succeeding and said so. In a shell pipeline the exit status is the status of the last command, and the pager exits zero whatever it is given. The build had failed, several hundred lines above what I was looking at.

I corrected it in the same session, which is the only reason it did not become a fact somebody else planned around. That is the cost to you: a wrong “done” turns into a decision made on a false fact.

What was actually going on

The thing measured and the thing reported were different, and nothing in between complained. The same class appeared from every direction.

One round reported all twenty-five test suites green. Re-running them produced twenty-two green and three failures. The report was not dishonest: it described a run that had happened. It was simply not a verification, because nobody had checked it against the tree as it stood.

Another round left two lint errors in the codebase and finished without producing a report at all. Absence of a report reads as absence of a problem, and the errors were found by running the gate rather than by reading anything.

There was a second-order problem. Several people or processes working in parallel against one shared development database means a full test run measures the database as much as the code. A batch run showed two failures that passed when the same suites were run individually, because another process was writing to the same tenant at the same time.

What we changed

The habit is tedious and it works: run the gates yourself, at the end, against the tree you are about to hand over. Lint, type check, migrations forward and back, the test suites, the browser suite. The value is not catching anyone out. The report and the gate disagree often enough to be worth twenty minutes, and each disagreement is otherwise found by somebody downstream at a worse moment.

A full-suite result now carries its conditions. An uncontended run is a claim about the code, and a contended one is a claim about the afternoon. The reports that were worth reading all had the same contents:

• The command and its real output: suite names and counts, not the word pass.
• What was not verified. Several rounds of mobile work ended with an explicit statement that no screen had been driven on a device.
• Deviations from what was asked, including files touched outside the stated scope.
• Adjacent defects found and not fixed, logged precisely enough for someone to pick up.

What it did not fix

More and more work is done by processes that report on themselves. A generated summary saying everything passed is written with the same confidence whether or not it did, and it is fluent either way. A badly written report from a tired person carries signal in its raggedness; a generated one does not.

Re-running the gates does not remove that risk. It moves it to a check somebody has to remember to do. I do not exempt myself from this, and the first paragraph is the reason.

What to ask your own team or supplier

Put these to your supplier before you accept a report that says the build passed:

• When a status report says tests pass, which command produced that, and was it run against the code we are accepting?
• Does the report say what was not verified, for example no screen driven on a real device?
• Was the test run made while other work was touching the same environment?
• Who re-runs lint, type checks and migrations at the end, and is that written down?
• Where a pipeline or wrapper script sits between the tool and the result, which exit status is being read?

Where this ends up

A report is a claim about work and a gate is a measurement of it. That is the rule behind AI-first delivery: verification is a separate act from production, the evidence is the running system rather than the description of what was done, and a check that can fail is worth more than a report saying it passed.