Every question was correct and the whole set was broken

A ten-year-old could have scored full marks on a hundred and fifty questions by tapping the first option every time and never reading anything. Not because the questions were easy. Because every correct answer sat in position one.

This was content generated for a learning app that pays points for quiz performance, so the child was one pattern away from a payout with no learning attached. The questions were already published and already on her tablet.

Nobody caught it in review, and by review I mean my own. I had checked the questions. Each question was factually correct, each had four plausible options, each pointed at the right one. Reading them one at a time, there is nothing to find. The defect does not exist in any single question. It only exists in the set.

Why the bias was there at all

When a generator has a free choice that nothing constrains, it does not make that choice randomly. It makes the same choice repeatedly. Writing a question, the natural order is to state the correct answer first — that is the thing you know — and then invent three distractors. Nothing in the process ever says “now move it”. So the correct answer stays where it was written, in every question, forever.

That is not a quirk of one tool. It is what happens to any unconstrained degree of freedom in a generated artefact. The order of keys in a produced object, the choice of a placeholder name, which of two equivalent branches an emitter takes: if nothing specifies it, it will be constant, and constant is not the same as arbitrary. Arbitrary would have been fine. A quarter of the answers would have landed in each position by accident.

It was found by an independent review of a later batch, run before publishing rather than after, which is the only reason it was found before the child did. Two reviewers with different briefs — one on safeguarding, one on factual accuracy — both flagged it, neither having been asked to.

Fixing published content is a different problem from fixing a generator

The generator can be fixed going forward. The hundred and fifty questions already sitting on a device cannot be regenerated, because a child had already answered some of them and her scores refer to the questions as they exist.

So the correction had to move the answer index without changing which answer is correct. That is a narrow thing to get right, and it is very easy to write a migration that shuffles the options and leaves the index pointing at whatever now occupies the old slot. The child then sees a quiz where the marking is wrong, which is enormously worse than one where the marking is predictable.

The load ran in a single transaction with the check written against the meaning rather than the mechanism:

• Before touching anything, record the text of the correct option for every question.
• Shuffle the options and rewrite the index.
• Re-read every question and compare the text now sitting at the new index against the recorded text.
• Commit only if every one matches. Otherwise roll the whole thing back.

The important detail is that the assertion is on the string, not on the number. The number is the thing being changed, so an assertion about the number cannot distinguish a correct migration from a broken one. When a migration’s whole purpose is to change a value, your invariant has to be stated in terms of something that is not allowed to change. In a data migration the thing that must not change is almost always the meaning, and the meaning is almost always in a different column from the one you are editing.

What replaced hand-checking

The next batches were two hundred questions and forty-eight questions. Hand-tracking the spread across four positions over two hundred items is a task that is guaranteed to fail somewhere in the middle.

So the answer position stopped being a thing anyone chose. The questions are still written with the correct answer first, because that is the natural way to write one. A deterministic rotation then moves it into a fixed pattern per quiz, so the even spread across four positions is produced by the process rather than maintained by attention.

The load then refused to commit unless it observed all of this at once: the expected count of lessons, quizzes and questions; every answer index inside the bounds of a four-option array; an even spread across the four positions within each quiz; and no duplicated option text on any question. Any one of those failing rolls back everything.

Sixteen answers were then read back by hand, not to check the rotation worked — the transaction already proved that — but to confirm the index and the text had moved together, which is the one failure the transaction’s own arithmetic could conceivably have shared a bug with.

Two things this changed in how I think about content

Content is data, and data gets migrations, transactions and invariants. It is tempting to treat authored material as a soft artefact that gets reviewed by a human and typed in. Once it is a hundred and fifty rows with a foreign key and a numeric index into an array, it has all the failure modes of any other data, and none of the tooling unless you build it.

Verify the distribution, not the items. Every review process I have run over generated material reviews items. Item review is necessary and it is structurally blind to the class of defect that lives in aggregate. The questions to ask of a generated set are the boring statistical ones. Are the answers spread across the positions? Do the same three distractors keep reappearing? Is one option consistently the longest, which is its own well-known tell? None of those are visible at the item level, and all of them are visible in about four lines of query.

The honest limit is that this only catches the biases you thought to measure. An even spread across four positions says nothing about whether the correct option is systematically the most detailed one, and I have not checked that. What I would say with confidence is that any generated set has at least one constant where you expected variety, and the cheapest way to find it is to count things rather than read them.