Our new website told Google to go away on every page

You launch the new website, and within weeks you have disappeared from Google. The agency checks and finds nothing wrong, because there is nothing wrong that you can see: the pages load, the words are right, the forms work. Recovering is measured in months, and in that time the people who would have found you do not.

We came within one command of doing this to ourselves. The dangerous failures in a publishing process are the ones that produce a working site. A build that crashes gets fixed in minutes. A build that succeeds, goes live, looks perfect and quietly tells every search engine to forget the domain runs for weeks before anybody notices.

What was actually going on

Our site can be built for two addresses: the private staging address we used while rebuilding, and the real domain. Staging must never be indexed, because two copies of the same content competing is worse than neither being found. So the “do not index” instruction was worked out from the address, which is the right instinct. It also said: if no address is set at all, treat it as unsafe and block everything. Sensible on its own.

The trouble was the ordinary, documented build command, which set no address. The system fell back to the staging address. So the normal command produced a site with “do not index” on all 245 pages, and each page claiming staging as its proper home. Nothing about that build looks wrong. It finishes, the page count is right and the content is correct. The only difference was a short line in the head of every page.

It was caught by a manual check, not a test. The deploy was about to replace a live site, so someone looked in the finished files for the robots instruction. It was on every page, including the home page.

We found two more of the same kind in the same hour. A copy of the staging instructions file, kept while the real one was written, was sitting in the folder that is published as it stands. So a page on the real domain was telling crawlers to stay away from everything. No engine would have honoured it at that address, but a file on your real domain telling crawlers to leave is not helping.

The third is the one to worry about if you write your own checking tools. We had built an audit that walks the finished site and reports pages with too little content. Later we taught it a correct rule: skip pages marked do-not-index, since a page that cannot be found cannot be judged on whether it would be found. Run against a staging build, it reported 245 pages and none below target. A perfect score, produced by a rule that was right in itself, on a site where every page was blocked. The all-clear meant the opposite of what it said.

What we changed

There is now a separate production build command that states the address explicitly, in a form that behaves the same on every kind of computer. The audit refuses to produce a report at all if every page is marked do-not-index, and prints how to rebuild properly. A refusal cannot be scrolled past; a warning can.

Checks run against the finished files before anything is uploaded, and again on the server against the staged copy before the swap. Any failure stops the deploy. The home page must not contain “do not index”. The proper address must be the real domain. Nothing may mention the staging address. The feed and the key file must exist. They take under a second.

What it did not fix

The checks tell you what is in the files you are about to publish. They do not tell you that search engines have since accepted them, which takes days and is measured elsewhere. And each check exists because of something that happened, so a fourth kind of slip has no check until it happens.

The pattern, for anyone launching a website

Each of the three faults was a safe default that turned unsafe when it was all that stood between you and going live. Block-unless-told-otherwise is right for staging and dangerous as a launch default. Publish-everything-in-this-folder is right for images and wrong for the note you left there. Skip-what-cannot-be-found is right for a report and wrong when the answer is everything.

None would have thrown an error, and all would have been found by the same question: what is actually in the files I am about to publish? Ask your builders to show you that, on launch day, as a fact in the file and not as a belief about a script.

Where this ends up

That distinction is not particular to a website. It is what the cutover checks in the work described under enterprise modernisation rest on: correctness proved against the finished result on every rehearsal, not against the process that produced it.