The quiz score was a fine proxy for reading, except on the lessons that mattered
The question was simple and I could not answer it: has the child actually read the lesson, or did she go straight to the quiz?
The learning app has a reader screen showing a written lesson, and a quiz attached to it that pays points. The parent wanted to know whether the reading was happening. I assumed the answer was in the database somewhere and went to find which column it was.
It is not in any column. Nothing anywhere records that a lesson was opened.
The progress table is keyed on the child and the quiz, and every column on it describes quiz performance — score, attempts, when it was last taken. There are four routes the child’s device can call, and none of them is a mark-as-read. The route that starts a quiz attempt validates that the lesson is assigned to that child and that it is published, and it does not check, because it cannot, whether the lesson was ever displayed.
So the honest answer to the parent was: nobody knows, and the system was never built to know.
The signal existed one layer below where it was needed
The interesting part came next. The reader screen draws a progress bar as you scroll, and it restores your position when you come back to a part-read lesson. To do that it computes a live scroll fraction and persists it.
On the device. In local storage. Never sent anywhere.
The exact number I had spent an hour concluding did not exist was being calculated continuously, written to disk and thrown away, because it had been built for a presentation feature rather than as a fact about the child’s activity.
That is a pattern I have hit more than once and now look for deliberately. Before designing a way to capture a signal, search for the places where the system is already computing it for a local purpose. Scroll positions, cache timestamps, retry counters, the “last seen” value a UI uses to draw a badge — these are usually computed accurately, close to the event, and discarded, because whoever wrote them needed them for one render and had no reason to think anyone else cared. Turning one of those into a recorded fact is much cheaper than instrumenting from scratch, and it is more accurate, because it is already load-bearing for something the user can see.
Why the obvious proxy was worse than it looked
The available proxy was the quiz score. If she scored well, she must have read it. That is a reasonable-sounding argument and it is wrong in a specific way that only shows up when you look at it per subject rather than on average.
On a maths lesson, the proxy holds. You cannot guess seven hundred and twenty divided by eight from a list of four numbers. The score is evidence of something.
On a safety lesson, the proxy collapses entirely. Every question of the form “what should you do if…” has one option that sounds like the careful, sensible answer, and a ten-year-old identifies it from the list without having read a word. She could score full marks on a subject she never opened.
So the proxy is strongest on the content where it matters least — arithmetic practice, where not reading the lesson costs her a bit of learning — and useless on the content where it matters most, where the whole purpose of the lesson is that she has actually absorbed something before she needs it.
A proxy metric’s accuracy is not one number. It is a distribution across the domain, and the useful question is where it is worst, not what it is on average. I have seen the same shape in operational metrics: a health check that is a good proxy for service health except during the exact failure mode you introduced it to detect, because that failure mode is the one where the check’s dependency is also down.
What was rejected, and why
The cheapest option is a minimum dwell time. If the reader was on screen for ninety seconds, count it as read.
Rejected on two grounds. It is trivially gamed — leave the tablet open and go and do something else, which a child works out immediately and which cannot be distinguished from reading. And it punishes a fast reader, who genuinely finished and is told she has not. A measure that both a cheater and an honest fast reader fail is not measuring reading.
The scroll fraction is better and still weak. It shows the content passed under the eyes. It shows nothing about attention.
The route that turned out to be most useful was not instrumentation at all. It was writing the questions so that answering them requires having done the thing. A later set of exercise-based lessons asks what ended up in a particular cup after a swap, how many times a repeating chunk occurred in a physical layout, and where an item lands under a top-to-bottom rule match. Those answers do not exist anywhere except in the activity. They are a weak but real check, and they cost nothing to build because they are just how the questions are worded.
The limit I have not solved
None of these verify understanding. They verify exposure, at best. For a safety subject that distinction is the whole thing, and no amount of engineering closes it — the real verification is a parent having a conversation, and the most I could do was write the lessons so they give a parent openings for one.
I record that as an honest limit rather than a roadmap item, because I think it is the correct conclusion and not a gap. The instinct when a product cannot measure something is to build a measurement. Sometimes the right answer is that the measurement is not available to software, and the product’s job is to make the human check easier rather than to substitute for it.
Two things I will carry forward regardless. Search for the signal you already compute and discard, before you build a new one. And when you accept a proxy, name the part of the domain where it fails before you ship it — because if you do not, you will discover it in the one subject where being wrong actually costs something.