A child was served a quiz nobody had written
In a children's learning app, a parent published a seven-question lesson and the child got one question. Two faults combined, and a third was found beneath them.
A parent accepted seven AI-generated questions onto a lesson in a children’s learning app and published it. The child opened the lesson and was given a quiz with one question in it.
The data explained it straight away. That lesson had six quizzes attached to it, created within six-tenths of a second of each other, holding one question each. A child had been served content that nobody had created. Two faults, each survivable on its own, had combined, and the read path showed no error.
A result that looks plausible and is wrong is what a supplier’s “tested and working” report cannot see. Here, every step did something reasonable on its own.
What was actually going on
The first fault: a stale value in a loop. Adding the accepted questions ran through a helper whose job is to make sure a quiz exists before adding a question to it: create one if there is none, otherwise use the existing one. The helper read the quiz identifier from component state, and in this framework state is fixed for the duration of a render however many times you set it.
The loop ran seven times. Each pass read the same captured value, which was empty, decided there was no quiz and created one. Six quizzes were created, each with one question. The seventh question landed on one of them. Every pass was correct in isolation, and the “otherwise use the existing one” refers to something that changes during the loop, while the loop was reading a snapshot taken before it started. Any get-or-create in a loop, where the get side is served by something refreshed between iterations, creates N rows.
The second fault: a query with no ordering. Six quizzes on a lesson would have been messy but visible. What served wrong content was that the child’s reader takes the first quiz from a list, and that list came from a query with no ordering clause. With one quiz, an unordered query returns the right answer every time, so there is nothing to notice. With two or more the answer is whatever the storage engine finds convenient, and that can change after a rewrite, an index change or a restore.
The whole content path, from subject down through lessons and quizzes to questions, was then checked for the same fault. Exactly one edge was broken: the middle one, with neither a stored sort order nor an explicit ordering.
What we changed
Turning one quiz per lesson into several forced two decisions that had been hiding in constants.
Points were a flat value per quiz. With six quizzes a lesson’s payout multiplies by however many hang off it, so a parent splitting one long quiz into three would triple what it pays without seeing a number. Each quiz now carries its own price, chosen when it is created.
Progress was keyed on the child and the lesson. That cannot distinguish one quiz finished out of six from all six finished. Progress is now keyed on the quiz, with the lesson kept as an indexed column so the roll-up still works.
What it did not fix
A related ordering fault in the child’s lesson list is still open. Lessons have no sort order column, and the device orders them by last-updated time, newest first. A child was taught three-digit arithmetic before single-digit arithmetic, and a later batch of lessons came out scrambled for the same reason.
The stopgap staggers the update timestamps so the order comes out right. It works today and is not a fix: any later edit bumps a lesson’s timestamp and sends it to the front of the list, so the order breaks the first time somebody corrects a typo. The real fix is an explicit sort order column, which is a schema change and was not that day’s work. The reason the stopgap is wrong is written down beside it.
What to ask your own team or supplier
- When a relationship changes from one-to-many, who checks every place that takes the first element?
- Is every query that feeds a screen ordered explicitly, or does it rely on the database returning rows in a convenient order?
- Is any constant, such as a price or points value, standing in for something that should belong to each item?
- Where a stopgap is in place, is the reason it is wrong written next to it?
Where this ends up
Cardinality changes are not additive: going from one to many invalidates every assumption that a single element let you avoid stating. Checking for that, instead of reporting the change as tested, is the verification work described under AI-first delivery.