Memory, Evaluation, Guardrails

The first thing I do is refuse to treat memory as one thing. "Memory" is four different problems that people give one name, and they have different storage, different lifetime, and different failure modes.

One: what did we just say — the last few turns of the conversation. Two: what has happened in this whole session, which must survive a server restart. Three: what do we know about this user or this entity that should outlive the session. Four: what have we already answered before, so we do not pay to compute it twice.

So I design each one separately, and for each I write down three things: where it is stored, how long it lives, and what happens when it is wrong. The last one matters most. A wrong memory is worse than no memory, because the system states it with full confidence.

I split them by who produced the content. Retrieval reads from a source of truth that someone else owns — in my project, the published annual reports. Memory reads things my own system produced or inferred.

That line matters because the trust level is different. A number from an annual report is a fact I can cite with a page number. A number from my own earlier answer is a claim, and it may have been wrong when I made it.

So retrieved content can go straight into the answer with a citation. Remembered content has to be labelled, and I have to be willing to throw it away and retrieve again.

The context window is the maximum amount of text the model can read in one call — roughly a few hundred pages for a modern model, but it is a hard ceiling, and everything you put in it costs money and latency.

The naive approach is to send everything: full conversation history, all retrieved documents, all memory. That breaks in two ways. It hits the ceiling, and long before that it degrades quality, because the important sentence is buried among thousands of irrelevant ones.

So I treat context assembly as an explicit, ordered budget, not a concatenation. Highest priority is the current question. Then the evidence I actually retrieved for it. Then a short summary of the older conversation. Memory hints go last, and they get dropped first when space runs short.

The key point is that this ordering is a design decision I can defend and test — not whatever the code happened to append last.

The simplest method is a sliding window: keep the last N messages, drop the rest. It is one line of code and it is wrong for our case, because the user says "compare that with Rio" and the thing being compared was dropped.

What I use instead is a summary buffer. When the history goes past a threshold, I take everything except the most recent few messages, ask the model to summarise it in two or three sentences, delete the originals, and insert the summary in their place.

So recent turns stay word-for-word, because that is where pronouns and follow-ups point. Old turns survive as a compressed trace instead of disappearing.

The important safety rule: if the summarisation call fails, I do nothing and keep the original messages. A failed compression must never delete history.

A normal cache matches the exact string. That is useless for natural language, because "BHP revenue in FY2024" and "How much did BHP earn in FY2024?" are different strings asking the same thing.

A semantic cache fixes that by comparing meaning. I turn the question into an embedding — a list of numbers that represents its meaning — and compare it with the embeddings of past questions. If the similarity is high enough, I reuse the old answer.

It helps a lot in a domain like this, because a handful of questions cover most of the traffic and each miss costs several model calls plus retrieval.

Where it hurts is the near-miss. "BHP revenue FY2024" and "BHP revenue FY2023" are almost identical as sentences but completely different as questions. A cache tuned too loose will confidently serve the wrong year. That is why I do not use one threshold — I use two.

This is coreference — the sentence points at something mentioned earlier instead of naming it. If you send it to retrieval as-is, you search for the word "dividends" with no company and no year, and you get noise from five companies.

There are two ways to fix it. You can ask a model to rewrite the question using the history, which handles anything but costs a call and can hallucinate a company that was never mentioned. Or you can carry forward the specific entities you care about, which is cheap and predictable but only covers the entities you listed.

I chose the cheap one, because in this domain the ambiguity is almost always the same two slots: which company, and which financial year. I scan the recent messages for a ticker and a year, and if the current question is missing one, I append what I found.

It is deliberately narrow, and I would defend that: it is deterministic, testable, adds no latency, and if it finds nothing it changes nothing.

Four layers, and every request passes through them in a fixed order before any retrieval happens.

First, durable session state. The whole graph state is checkpointed to PostgreSQL after every step, keyed by a thread id. So a conversation survives a server restart, and I can replay exactly what the system saw at each step — which is as much an audit feature as a memory feature.

Second, context compression. Once the history passes ten messages, the older ones are summarised into a single note and the last four are kept verbatim.

Third, entity carry-over. I pull company tickers and financial years out of the recent messages and top up the current question if it is missing them.

Fourth, the semantic answer cache. Past question-and-answer pairs live in their own vector collection, separate from the report chunks. Above 0.95 similarity I return the stored answer and skip retrieval entirely. Between 0.80 and 0.95 I run the full pipeline but seed it with the old answer as context.

Because at lookup time all I have is a question. If the stored vector represents an answer, I am comparing a question against an answer, and those two texts look different even when they belong together.

A question is short and interrogative; an answer is long, full of numbers and hedging. Their embeddings sit in different regions, so similarity scores get noisy and unreliable.

So I store the question as the searchable content and keep the answer and its sources alongside it as metadata. That makes it question-to-question matching, which is the comparison I actually want.

The write is gated, not automatic. An answer is only saved if two conditions hold: the quality grader passed it, and the groundedness check did not come back as "no".

Groundedness means every fact in the answer could be found in the documents that were actually retrieved. If that check fails, the answer still goes to the user — with a visible warning — but it is never written to memory.

There is one more detail I am quite pleased with. The warning text is appended after the save, so what gets cached is the clean answer. Otherwise a future user would receive a stale caveat about a verification run that has nothing to do with their question.

The general principle: the read path is optimistic, the write path is pessimistic. Getting a cache miss is cheap. Poisoning the cache is not.

The first move is to stop asking "is the answer correct" and start asking "how does this system behave across a fixed set of situations". One run tells you almost nothing, because the same input can produce different output.

So I fix everything I can. A fixed set of cases with known expectations. Deterministic checks wherever a check can be deterministic. And where I have to judge something subjective, I score a property of the answer rather than matching the text.

The second move is to evaluate components before the whole. If I only test end to end, a failure tells me the system is wrong but not where. In my project I can test the router alone, retrieval alone, and the grader alone, so a regression points at one component.

And then there are two phases, not one. Offline evaluation is a fixed test set before release — it stops known regressions but cannot tell you about real users. Online evaluation is measuring live traffic after release — that is where reality arrives. You need both, and the offline set should be fed from what online finds.

I break the pipeline into stages and measure each stage on its own terms, because "answer quality" is not one property.

Routing: did the system decide correctly whether to retrieve, to search the web, to ask a clarifying question, or to just answer. That is a classification problem with a clean label, so it is cheap and exact.

Retrieval: given a question with a known correct figure, did a document containing that figure reach the answering step, and at what rank. That is measured without any model in the loop.

Groundedness: is every claim in the answer supported by the documents that were retrieved. This is the hallucination metric and it is the one I care about most in a regulated setting.

Boundary behaviour: on a question the system should not answer, does it decline properly instead of inventing something. And alongside all of that, latency and cost per query, because an accurate answer nobody waits for is not a product.

Three things: isolate the change, compare on the same cases, and be honest about how much the sample size can support.

Isolate first. If I change retrieval, I take the model out of the loop entirely and score retrieval directly. Otherwise the model's own variance is larger than the effect I am trying to see.

Then compare paired, not averaged. Same cases, both configurations, and I look at what moved per case — how many got better, how many got worse, how many were found that were previously missed. A mean can improve while individual cases regress, and those regressions are what an incident is made of.

Then I state the limit out loud. With about a dozen cases I can see direction, not significance. I would call that evidence to proceed, not proof — and if the decision were expensive, I would go and label more cases before shipping it.

Offline means a fixed set of cases with expected outcomes, run on demand. It is repeatable, cheap, and safe, so it is the right thing to gate a release on. Its weakness is that it only contains failures you already thought of.

Online means measuring the system on real traffic. Usually that is an A/B test: some share of requests go to the new version, the rest to the current one, and you compare outcomes. Its strength is that it finds the questions you never imagined. Its weakness is that some users get the worse version while you learn.

The two connect in a loop, and that loop is the actual methodology. Online finds a new failure, that failure becomes an offline case, and the offline suite gets stronger. If your test set only grows when an engineer has an idea, it will drift away from what users do.

Before a real A/B I would run in shadow mode — the new version processes real traffic but its output is logged rather than shown. You get real distributions with zero customer risk.

That is the honest weak point of LLM-as-judge, and the answer is that the judge is a component like any other, so it needs its own evaluation.

Concretely: I take a sample of cases, label them myself, and measure how often the judge agrees with me. If agreement is poor, the judge's verdicts are not evidence and I should not be shipping on them.

Then I make the judge's job as narrow as possible. It never scores "quality" on a scale. It gets the question, the retrieved documents, and the draft answer, and it returns a strict structure — pass or fail, plus a list of claims it could not verify. A constrained output is far more stable than a free-text opinion.

Two more things that matter. I tell it explicitly which differences do not count as unsupported — different units, rounding, accounting notation — because otherwise a correctly grounded answer gets flagged constantly. And I set its strictness deliberately: for pass-or-fail I told it to be lenient, because that gate only exists to catch answers that are empty of content.

By splitting it into tiers by cost, and running each tier at the frequency it can afford.

The cheap tier calls no external service at all. It feeds synthetic outputs through the real evaluator code. That does not prove the system works — it proves my cases are well-formed and my scoring logic is correct. It is free and fast, so it can run on every commit.

The middle tier runs the real components, but only a handful of cases, and only the cheapest path that is still meaningful. For routing I run the routing nodes alone rather than the whole graph. That is a smoke test on a pull request.

The expensive tier is the full suite against real services and real indexes. That runs on a schedule, not on every push, and its results are compared against a stored baseline so a drop is visible.

The rule I would set as a standard: the release gate must be deterministic. If a build can fail because a model felt different today, engineers will learn to re-run until green, and the gate stops meaning anything.

Four suites plus a separate retrieval harness. Routing, retrieval, calculation, and boundary behaviour, with about forty cases in total as line-delimited JSON files.

Every case has an id, the query, the expected outcome, and a why_it_matters line explaining what regression it would catch. The loader refuses to run a case that is missing any of those, so the suite cannot quietly fill up with cases nobody understands.

There are two run modes behind one command line. Mock mode calls nothing external and checks that my cases and evaluators are correct. Real mode runs the actual nodes, and I can scope it down to one suite, a case count, or a single case id, which is what keeps real-mode cost controllable while debugging.

One design choice I would highlight: real mode runs the cheapest path that is still meaningful. Routing cases execute the routing and query-rewrite nodes directly rather than the full graph, so a routing regression is caught without paying for retrieval and synthesis.

My keyword retriever was matching words literally, so "Revenue" with a capital R and "revenue" were two unrelated terms, and "revenue," with a comma was a third. I wrote a proper tokeniser to normalise them.

While reading the library source I found something I did not expect. BM25 is supposed to handle common words automatically: a word appearing in most documents gets a low weight, so "the" should be nearly worthless. The formula can produce a negative weight for such words, and this library refuses negative weights — it clamps them up to a floor instead.

The result inverted the intent. "The" appeared in 84 per cent of my chunks, so its true weight was negative, and the floor lifted it to 1.81 — higher than "Group" at 1.54, which is a genuinely informative term. Only eight words were affected, but those eight appear in nearly every natural-language question.

I measured it. For the query "what was the revenue of the group in 2024", 59 per cent of the top document's score came from "what", "was", "the", "of" and "in". After removing those words, the new top five documents had zero overlap with the old top five. That is not a nudge in the ranking — the results had been decided by filler words.

The lesson I took is the one I would bring to a team: the textbook claim was right about the algorithm and wrong about the implementation, and only reading the source and measuring the actual scores showed the difference.

I built a small A/B harness that calls the retrieval node directly and never touches the language model, so the only thing that differs between the two runs is the retrieval configuration.

The ground truth is exact figure strings that I verified exist in the ingested text — for example 55,658 for BHP's FY2024 revenue and 29,016 for its underlying EBITDA. A case scores as a hit if a returned document contains the figure, and I record the rank at which it first appears.

I report hit count within the final cut, hit count within the top five, and mean rank of the hits. Then per case I print whether it improved, regressed, was newly found, or was newly lost — because the aggregate can improve while individual cases get worse, and the regressions are what I actually want to see.

The two configurations were the old one — a candidate pool of eight, no reranking — against the new one, a pool of twenty followed by a cross-encoder rerank down to eight.

One practical detail: the reranker is a paid API with a token-per-minute limit, so the harness paces itself with a fixed pause between cases. Without that, the comparison fails for rate-limit reasons and looks like a quality result.

As layers at boundaries, not as one filter at the end. A single check before the answer is the design people reach for first, and it is too late — by then you have already paid for the work and you have nothing left but to block.

I put a check at each boundary the request crosses. At the input: is this question in scope, and is it clear enough to attempt. At retrieval: is what came back actually relevant, and did I find something for everything the user asked about. At generation: is every claim in this draft supported by the evidence. At output: is there a substantive answer here at all, and do the citations resolve.

Two rules I hold across all of them. Cheap and deterministic before expensive and probabilistic — if a length check or a lookup can catch it, do not spend a model call. And every layer states its verdict into the run state, so the decision is recorded rather than implied by which branch the code took. That recorded trail is what makes the system auditable.

It depends on the layer, and the important thing is that I decided it per layer, deliberately, rather than letting the exception handler decide.

My rule is this. If the layer is a precision improvement — it makes a good answer better — it fails open, because losing it degrades quality but blocking on it takes the whole service down. If the layer is a correctness floor — without it a wrong or empty answer reaches the user — it fails closed.

So in my project the two model-based judges both fail open. If the relevance grader errors, I pass all the retrieved documents through instead of none. If the groundedness grader errors, I mark the answer as passing rather than blocking it. Both are optional precision layers sitting on top of retrieval that already worked.

The final check is different. It is deterministic, it makes no network call, so it has essentially nothing to fail on — and it fails closed by construction: no substantive content means the fallback path, always.

What I would add in a bank is that fail-open must be loud. Silently degrading is how a system runs for a week with its safety layer switched off. Every fail-open needs a metric and an alert.

I would separate three things that get bundled together: can the user see why, can an engineer reconstruct what happened, and can a reviewer prove it months later.

For the user, explanation has to be in the answer itself. Every figure drawn from a report carries a citation marker that resolves to a company, a financial year and a page — so the claim can be checked against the source. And when the verification step could not confirm something, the answer says so, in the answer, not in a log.

For the engineer, every decision is written into the run state and the state is checkpointed at every step. So I can reconstruct not just the final answer but which route was chosen, which documents survived grading, what the judge said. Reconstructable is stronger than logged, because logs are prose and state is structured.

For the reviewer, the gap in my project is that this trail is operational storage, not an audit record. It is not immutable and it has no retention policy. In a regulated environment I would write the decision trail to append-only storage with a defined retention period, and treat it as a different artefact from the conversation state.

I would push back gently on the word "stop". A language model predicts plausible text; producing a confident wrong number is not a bug you can patch out. So the goal is not prevention, it is detection and containment.

Detection works because in a retrieval system I have something to check against. I pass the draft answer and the exact documents it was given to a verification step, and ask a narrow question: which claims in this answer cannot be found in these documents.

Containment is what happens next, and it has three settings. If a lot is unsupported, I retry retrieval targeted at the specific claims that failed. If a little is unsupported, the answer still goes out, but the user sees which claims are unverified. If nothing is unsupported, it goes out clean.

One refinement worth mentioning: the verifier also classifies why a claim is unsupported. Was it arithmetic derived from figures that were present, was it estimated from a trend, or does it have no connection to the documents at all. The first is often legitimate and just needs to be shown as a derivation. The last is the real hallucination. Treating those three identically loses information the user needs.

Three different situations, and mixing them up is a product failure rather than a technical one.

Refuse when the request is outside what the system is for. Somebody asking for a poem from a financial report assistant should get a clear boundary and a statement of what it does cover — not a poem, and not a hedge.

Ask when the request is in scope but underspecified. "What were the numbers last year" is a reasonable question with a missing subject. Guessing produces a confident answer about the wrong company; asking one short question costs a turn and gets it right.

Answer with a caveat when I have real evidence but could not verify all of it. Refusing there throws away a mostly useful answer; hiding the gap is dishonest. Showing the answer and naming the unverified parts respects the user as someone who can decide.

And the caveat has to be specific to be useful. "This may be inaccurate" on every answer is noise. "This figure could not be verified against the retrieved documents" on the one claim that failed is information.

Three constraints. A hard cap, a changed input, and a defined outcome when the retries are exhausted.

The cap is the easy part — a counter in the state, and no more than two retries. Because the state is the same object the whole graph reads, there is one counter, not one per branch.

The second is the one people miss. A retry that sends the same input gets the same answer and has only spent money. So every retry has to change something. In my case the retry is targeted: the claims that failed verification become the new search terms, and the companies whose claims failed are marked as not-yet-found so the next round searches specifically for them.

The third is that running out of retries is a normal outcome with a defined path, not an error. It goes to the fallback response, which explains what the system covers. An unbounded loop and an unhandled exhaustion are the same bug seen from two ends.

Six checkpoints, in the order a request meets them.

One, scope. The routing step classifies the question, and anything unrelated to the five companies gets a fixed reply that states what the system covers. Two, clarity. If the question is too vague to retrieve for, the graph asks a clarifying question instead of guessing.

Three, document relevance. Retrieved chunks are graded, and things like governance boilerplate or auditor's reports are dropped unless the question was about them. This step also corrects the per-company found-or-not flags, because a company is only really found if it has a document that survived grading.

Four, groundedness. The draft answer is checked claim by claim against the documents it was given, which produces the unsupported list and the retry signal.

Five, the final gate. A deterministic check that there is substantive content at all — and it sits on every path to the answer, including the direct-answer path that never touched retrieval. Failure routes to a fallback that explains the coverage.

Six, output shaping. Uncited sources are removed and markers renumbered, and a confidence level derived from the groundedness verdict is attached, with a named caveat when it is not high.

When the verifier reports an unsupported claim, it is instructed to write a short label of at most eight words that names the metric and the context — for example "RIO FY2025 revenue" — and explicitly not to reproduce the sentence or the number.

The reason is that the label is shown to the user in the warning. If the verifier echoed the full claim, the warning that exists to flag a possibly invented figure would print that figure again, in the answer, as text. The safety message would become another place the wrong number appears.

A user skimming might read the number and miss the framing. So the warning says which claim to distrust without restating the value they should not rely on.

It is a small thing, and it is the kind of thing I look for: not whether the check works, but what the check's own output does when it reaches a human.

Three places, and each was a decision rather than an accident.

The relevance grader fails open: if that model call errors, I pass all retrieved documents through rather than none. Losing the filter means a noisier context. Failing closed would mean no context at all, which turns a degraded answer into no answer.

The groundedness grader also fails open: on error the answer is marked as passing. That is the one I am least comfortable with, because it is the hallucination check, and its failure is silent. What makes it defensible is that the deterministic final gate still runs, and what would make it acceptable in production is an alert on the error rate — which I do not have yet.

The reranker fails open too: on a network error or a rate limit, the results keep the ordering from the fusion step. It is a precision improvement on a ranking that already exists, so degrading to the previous ordering is exactly right.

And one that fails closed on purpose: the final content gate is deterministic with no network dependency, so an empty answer always routes to the fallback. The floor is the layer that must not have an availability failure mode, which is precisely why it does not call anything.