An in-app assistant that cannot call out, because the index is the customer's entire pricing model
The knowledge base an in-app assistant would need to answer questions about a bespoke quoting system is the customer’s complete commercial position. Every pricing rule. Every material rate. Every labour charge. The relationship between cost and selling price on each of them.
For a fabrication business, that is not data about the business. It is the business. It is what a competitor would need to underbid them on any job, and it took decades to accumulate.
So the first design constraint was not accuracy or latency. It was that this index does not leave the machine it is computed on, which rules out every hosted inference and embedding service and makes everything afterwards harder.
What that constraint costs
The application runs on a modest shared virtual server: a couple of cores, no accelerator of any kind, single-digit gigabytes of memory, no swap configured, and other unrelated applications already resident on the same host.
Language model inference on that is slow. Not marginally — a generated answer of any length takes tens of seconds, and the request path in front of the application has a two-minute ceiling before it gives up. There is no budget for a large model, and no budget for a long context.
That is the trade, stated honestly. Hosted inference would be faster, cheaper to operate and better at the answers. It is unavailable because of what the index contains, and the correct response is to design around the constraint rather than to quietly relax it.
Retrieval before generation
The design that makes it viable is to treat generation as the last resort rather than the mechanism.
Most questions a user asks an assistant in this product are structural. What does this rule code do. Which products use it. What material group does this slot draw from. Which rate does this identifier resolve to. What steps does this calculation run.
Every one of those is a database query. The answer is exact, instant, and requires no model at all. It is only the open-ended questions — how do I set up a new product, why might this price be coming out low — that need language.
So retrieval is layered. Structured lookup against the actual entities first, then full-text search over the documentation and prior answers, and vector similarity as an optional third layer rather than as the foundation. Generation runs on what those layers return, when it runs at all.
That ordering is worth adopting even where compute is free. A retrieval assistant over a system that has a schema should query the schema before it embeds anything. Vector similarity over a document describing a rule is a lossy route to a fact that a query returns exactly, and the exact route is also the one you can test.
Fail closed, explicitly
The most important line in the design is one sentence: if the local inference service is unavailable, the endpoint returns a clear error.
There is no fallback. Not to a hosted provider, not to a smaller hosted model, not “just for availability”. The whole point of the local constraint is that the index does not leave the machine, and a silent fallback path is exactly the mechanism by which it would leave the machine, on the day nobody is watching, triggered by an unrelated outage.
Fallbacks are added with the best intentions and they invert the property the system was built to have. If a capability exists to protect data, the degraded mode has to be absence of the capability, never the same capability with the protection removed.
Stating it as a design principle, in writing, at the top of the specification, is what stops it being added later by somebody solving an availability complaint in good faith.
Two isolations, not one
Two separate boundaries had to be drawn, and they are easy to collapse into each other.
The first is tenancy. Every piece of indexed knowledge and every conversation is scoped to the company it belongs to. That is the same isolation the rest of the application has, applied to a new kind of data, and it needs to be applied at retrieval time rather than at prompt-construction time — a filter in the query, not an instruction in the prompt.
The second is the write boundary, and it is the one specific to assistants. The system is designed to improve over time: approved question-and-answer pairs from real usage get indexed, so answers get better as the product is used.
That learning affects the knowledge index and nothing else. Pricing rules, rates and templates change only through the existing administrative screens, with their existing permission checks and their existing validation. There is no path by which a conversation modifies a price.
This distinction is where assistant features go wrong. An assistant that can read is a retrieval problem. An assistant that can write is an authorisation problem, and it inherits every permission question the rest of the application answers — except that its authorisation now depends on interpreting natural language, which is not a foundation anybody should put a pricing system on.
Keeping the assistant strictly read-only, with a separate curated index as its only writable surface, means the blast radius of a bad answer is a bad answer.
Where this stands
This is a specification, not a shipped feature. It is designed against a verified picture of the host it would run on, with the model sizes, the storage locations and the failure behaviour written down, and it has not been built.
I am reasonably confident about the structure — retrieval layering, fail-closed, the two isolations — and much less confident about whether the latency is acceptable in practice on that hardware. My honest expectation is that the structured and full-text layers would carry most of the value, and that the generated layer would be slow enough that users route around it. If that turns out to be true, the right response is to ship the retrieval and drop the generation, not to relax the constraint that made generation slow.
The rule
Decide what the index contains before you decide where the inference runs, because the first answer determines the second. If the index is the customer’s commercial model, the inference is local and everything downstream is a consequence of that.
Then write the degraded behaviour into the design as a principle, in words, before anyone implements it. Fail closed is a decision that has to be made once, early, by someone thinking about the data — not repeatedly, later, by someone thinking about uptime.