AI assistant said nothing was waiting; four things were

If your AI assistant says nothing is waiting when you know something is, the usual cause is how its tools are described and filtered, not missing data. The assistant picks which tool to call from a short description, so a question in your words can go to the wrong one, and a date filter can hide the oldest open items. The first check is to ask the same question in your words and in the system’s words, and compare the answers.

Here is how that looked in a children’s learning app, and what we changed.

You ask the assistant that sits over your own records a plain question: has anything new come in that I need to answer? It says no. You carry on, and the things waiting for you carry on waiting.

In this case the records belong to a children’s learning app, and the question was whether a particular child had asked for anything new. The assistant said no. Four reward requests were sitting unanswered, dated 6 September, 6 September, 7 September and 12 September. The answer it had given, that nothing had come in over the past week, was true, and it was wholly misleading.

The cost is not a wrong number on a screen. It is a child who asked for something and heard nothing, and a parent who was told there was nothing to do. Swap the household for a business and the question becomes which customer requests are still open. An assistant that answers “nothing in the past week” is worse than no assistant, because you stop looking.

What was actually going on

There were two faults, and neither was in the data. Every record was in the right place.

The first was in how the assistant was told what each of its tools does. It has eighteen of them, each with a short description that it reads to decide which one to use. The words “asked for” appeared in exactly one. So a parent who said “asked for” was sent to rewards and the assistant stopped there. A child also has a second way to ask, by proposing a goal, and that tool described itself as “savings goals”, which is not how a parent talks. Asked about goals by name, the assistant found one proposed that same morning. Asked in ordinary words, it never looked.

The second fault was worse. The rewards tool filters by date, and the assistant chose seven days. All four unanswered requests were older than that. A request nobody has answered does not stop mattering because it is old, yet the tool treated age as if it did.

What we changed

We added one tool with a single job: list everything waiting on the parent. It has no date filter and no days setting, on purpose, so the assistant cannot choose a window that hides the oldest items. It returns the reward requests and the goals a child has proposed together, each with how many days it has been waiting.

We checked it against the live system rather than assuming. For this one child it found five things waiting, the oldest for sixteen days. All four ways of phrasing the question reached the right tool, and a direct question about goals still reached the goals tool.

What it did not fix

The wording of the tool descriptions is still what decides where a question goes. We fixed the cases we found. The log does not record an audit of the other descriptions for phrasings a parent might use, so we do not claim one.

The assistant also still relies on the model choosing well. A tool with no date filter removes one way to go wrong; it does not remove the model’s freedom to pick a different tool altogether.

And it does not replace the approvals screen the portal already had. The assistant tells a parent something is waiting. The decision is still made in the place that was built for it.

How to test your own assistant

1. For any question that means “what is still open”, check that the answer never depends on a date range. Open items are defined by their state, not by when they arrived.
2. Ask “has anyone asked for anything?” and “is anything waiting for me?”, then ask the same thing in the system’s own vocabulary. If the answers differ, the assistant is being steered by labels you do not use.
3. Check the oldest open item by hand. A confident “nothing” cannot tell you what it failed to find.
4. Ask your supplier which tool each of your questions reaches, and how that was checked.

Where this ends up

Assistants that answer from a business’s own records, and the checks that stop them sounding sure when they are not, are the subject of our AI and data assistants work.