A website is not a web application, and app teams keep shipping one as the other

We build operational software — booking systems, compliance registers, pricing engines — and we staff engineers who do the same for other people. Over the last fortnight we also rebuilt our own website, which is a different kind of system, and the gap between the two turned out to be wider than we had assumed going in.

This is a note about that gap, because it is the thing app teams get wrong. Not through carelessness. Through correctly applying habits that are right for the system they usually build and wrong for this one.

The client is different, and everything follows from that

A web application has one class of consumer: an authenticated human with a session, on a browser you can broadly predict, who will make dozens of requests in a sitting and whose problems you can see in your error tracking.

A website’s most consequential consumer is a machine. It does not log in. It arrives once, takes what the server gives it, and leaves. It may not execute your JavaScript, and if it does, it will not wait long. It never files a bug. It does not retry because a screen looked wrong. And its opinion — whether your pages exist, what they are about, whether anything on them can be quoted — is formed from single requests you never see the outcome of.

That is a harder contract than an authenticated session, not an easier one, and it is where the assumptions diverge.

Rendering: the page has to exist before the JavaScript does

In an application, deferring content to the client is usually correct. The user is authenticated, the data is theirs, the shell can paint immediately and fill in.

For a website that reasoning inverts. Every crawler that matters reads the HTML the server returned. Some execute JavaScript, on their own schedule, with no guarantee and no feedback. So content that only exists after hydration is content you have made conditional on somebody else’s infrastructure decisions.

This is not an argument against frameworks. It is an argument about where the render happens. Our own site is static HTML generated at build time — 251 pages, about ten seconds — which is not a technology preference so much as a statement that the output is a file, and a file cannot half-load.

URLs are the public interface, and they have consumers you cannot see

An application’s routes are internal. You rename one, you update the callers, you ship. Nobody outside has a reference to it.

A website’s URLs are an API with an unbounded set of clients you will never meet: links other people built, directory entries, pasted addresses, and the memory of every search engine that ever crawled you. Renaming one without a redirect is a breaking change to a published contract.

We knew that, generated a map of 140 legacy paths from the previous site’s URL inventory, and asserted every target returned 200 before writing it. Then we served pre-cutover content for days anyway.

The map was keyed on the trailing-slash form, because that is how the old site wrote its canonical URLs. Engines hold whatever form they first encountered, and plenty of ours had no trailing slash. nginx adds one automatically when a real directory exists — which is why every live page worked and nothing looked wrong — but a retired path has no directory, so those requests fell through to a 404.

The engineering point is the one about status codes as semantics rather than as error handling. A 404 is not the absence of an answer. It is an assertion that the address is dead, so the crawler keeps the stale record it already has. A 301 transfers identity. We were returning the first while intending the second, and the symptom was a search result showing marketing copy for a company we had stopped being.

There is now a check that reads the map off the server and requests every entry three ways — as written, without the trailing slash, and with a query string appended — asserting a 301 to the expected target. 141 entries, 420 requests, seconds to run. The lesson is not “write redirects”. It is that a published contract deserves a test, and this one had never had one.

The build can silently remove the business from search

Here is the failure that has no analogue in application work.

Our site builds for two hostnames, and staging must not be indexed, so the noindex flag was derived from the hostname. Sensible. But the default build command did not set a hostname, so it fell back to staging — and produced a site that was noindex on all 245 pages, with canonicals pointing at the wrong origin.

That build completes. The page count is right. Every page renders correctly. Deployed, it would have looked entirely healthy while telling every search engine to forget the domain, and nothing in any dashboard would have said so.

An application has nothing equivalent. A misconfigured flag behind a login shows up as a broken feature within the hour. Here the feedback loop is weeks, the signal is a slow decline, and by the time it is visible the recovery is months.

So the deploy asserts against the artefact rather than trusting the process that made it: the homepage does not contain noindex, the canonical resolves to the production origin, nothing references the staging host, the feed and key files exist, the page count is over 200. Each of those exists because of something that happened, and each is a grep against a file rather than a belief about a script.

Internal linking is a graph problem, not an editorial one

We had 142 articles and 11 of them linked to another article. A reader who finished one had a contact block and nowhere to go, and the library had almost no internal structure — which matters mechanically, because internal links are part of how a crawler decides what to fetch and what is worth keeping.

Curating that by hand does not survive contact with a growing corpus. So it is computed at build: every article now carries three related posts, 408 links, derived from the text.

It was wrong twice first, in ways worth recording. Version one gave a flat bonus for a shared topic, and that bonus alone cleared the relevance threshold — so with 79 articles filed under one topic, the suggestions were effectively random. Version two counted raw word overlap, which ranks common vocabulary as highly as distinguishing terms, and matched an article about payment idempotency to one about steel grades because both contained the word “quotation”. Version three scores tf-idf across the whole article body and lets the topic break ties only between posts that already share subject matter.

That progression is ordinary information-retrieval work. It is also the point: this was a ranking problem with a measurable output, not a writing task.

You cannot measure this with the tools you measure an application with

Application health is latency, error rate, saturation. None of it tells you whether a machine that visited once took anything away.

What we use instead: the access log, grouped by user agent, which tells you which crawlers arrived and — more usefully — which addresses they still believe in, because those show up as 404s. Search Console URL inspection, which distinguishes crawled from indexed, two states that are easy to conflate and mean very different things. And, because it is the only direct measurement, asking the answer engines the question a buyer would ask and seeing what comes back.

That last one produced the sharpest result we got. On the same day, with the same robots policy and the same crawl access, one engine described our product accurately from the new pages, and another said it did not have enough public evidence to identify it. The difference was not access. It was whether there was a sentence specific enough to lift.

What this means for who builds your site

The conclusion we would draw is not that websites need specialists in a separate discipline. It is that they need the same engineering discipline, applied to a different contract: HTTP semantics treated as semantics, URLs versioned like an API, build outputs asserted rather than assumed, and the retrieval problem treated as a retrieval problem.

Every failure above was an engineering failure with an engineering fix — a status code, a build flag, a scoring function, a test. None of them would have been found by looking at the site, and none of them were about the words on the page.

If the people building your website do not think in those terms, the site will look correct and quietly underperform, and nobody will be able to tell you why.