Design Decisions in STAR Form

The system answers questions about five companies across three financial years, and every answer is dense with figures. I was changing prompts, retrieval settings and routing rules, and my only feedback was reading a few answers and forming an impression. Two problems with that. The same question does not give the same answer twice, so a single good answer proves nothing. And when something got worse, I could not tell which of the eight or nine steps had caused it.

Turn "it feels better" into something repeatable, cheap enough to run often, and specific enough to point at one component.

Built four suites as line-delimited JSON files — routing, retrieval, calculation and boundary behaviour — around 40 cases in total, each with an id, the query, the expected outcome, and a why_it_matters line saying what regression it catches.

Two decisions inside that worth naming. The loader rejects a case missing any required field, including why_it_matters, so the suite cannot silently fill up with cases nobody can justify. And a suite is one entry in a registry mapping a name to a case-loader and an evaluator function, so adding a dimension does not mean touching the runner.

A regression now points at a component instead of at the system. Failures print per case with the specific mismatch, and the process exits non-zero if anything failed — so it is already shaped like a release gate.

A real run needs API keys, a built vector index and a built keyword index, and it spends credits on every case. That makes it something you run occasionally and reluctantly — which is exactly the wrong property for a check that is meant to catch mistakes early.

Get something meaningful out of a run that costs nothing and needs no environment, without pretending it proves more than it does.

Added a mock mode that constructs a synthetic output shaped like a successful run — expected companies and years, synthetic documents containing the expected metrics, an answer with a citation — and then feeds it through the real evaluator code.

I wrote the limitation into the source comment rather than the README: mock mode is not meant to prove the retriever works, it proves the evaluator logic and the case files are wired correctly. Both modes sit behind one command with a --mode flag, plus --suite, --limit and --id so real runs can be scoped down to a single case while debugging.

A free, deterministic, offline check that catches malformed cases and broken scoring logic — the two failure modes that would otherwise make every real run untrustworthy. Real mode stays available for when the question is about behaviour.

A routing case asks one thing: given this question, did the system correctly decide whether to search the reports, search the web, ask a clarifying question, or just answer. Running the full graph to find that out also pays for retrieval across up to five companies, a reranking call per company, synthesis, and two grading calls — none of which the case is testing.

Test the decision without paying for everything downstream of the decision.

Real-mode routing cases call the routing node and the query-rewrite node directly, and assemble the output dict by hand. Retrieval cases invoke the retrieval subgraph rather than the whole application. The dispatch is one function that picks an execution path per suite.

One detail: the runner also derives a couple of fields from the query with regular expressions — financial years, metric keywords — because the graph state does not expose them as top-level fields, and I would rather derive them deterministically in the harness than add fields to production state purely to make evaluation easier.

A routing regression costs two model calls to detect instead of a dozen. That is the difference between an eval you run while iterating and one you run before a release.

I wanted to know whether widening the candidate pool and adding a cross-encoder reranker actually improved retrieval. Measuring that through final answer quality is close to hopeless: the model's own variance is larger than the effect, so a difference in answers tells you nothing about the retriever.

Compare two retrieval configurations so that the only thing differing between the runs is the retrieval configuration.

Wrote a harness that calls the retrieval node directly — no routing, no synthesis, no grading, no model call anywhere in the loop.

• Ground truth is an exact figure string verified to exist in the ingested chunks — 55,658 for BHP's FY2024 revenue, 29,016 for its underlying EBITDA. A case is a hit if a returned document contains that string, and I record the rank at which it first appears.
• Metrics: hits within the final cut, hits within the top five, and mean rank of the hits. Then per case I print the verdict — improved, worse, newly found, or newly lost — because an aggregate can improve while individual cases regress, and the regressions are what an incident is made of.
• Baseline by substitution: the reranker is swapped for a pass-through function to produce the "old" configuration, rather than maintaining two code paths that could drift apart.
• A cache had to be cleared between configurations, because the candidate count is baked into the keyword retriever when its index file is loaded. Missing that would have made the second run silently reuse the first configuration.
• The reranker is a paid API with a per-minute token limit, about 9,000 tokens per case against a 100,000 ceiling, so the harness paces itself with a fixed pause. Without it the comparison fails for rate-limit reasons and looks like a quality result.

A per-case comparison table where every movement is attributable to the configuration. The two configurations were a pool of 8 with no reranking against a pool of 20 + 20 reranked down to 8.

My keyword retriever was splitting text on whitespace and nothing else. So "Revenue" with a capital R, "revenue" in lower case, and "revenue," with a comma were three unrelated index keys. Obviously worth fixing.

The interesting part was the second question: should I also remove stop words? The textbook answer is no: BM25 already down-weights common words, so a stop-word list is redundant.

Decide whether the textbook claim held for the specific library I was using, rather than for the algorithm in the abstract.

Read the library source instead of trusting the claim. The weighting formula can produce a negative value for a word appearing in most documents — and this implementation refuses negative weights. It clamps every negative value up to a floor computed as a quarter of the average weight.

That inverts the intent. Then I measured the consequence rather than reasoning about it:

• the appeared in 84% of my chunks. Its true weight was negative — −1.626 — and the floor lifted it to 1.811. That is higher than Group at 1.542, a genuinely informative term.
• Only 8 words were affected. But those eight appear in almost every natural-language question.
• For the query "what was the revenue of the group in 2024", 59% of the top document's score came from what, was, the, of, in.
• After removing them, the new top five documents had zero overlap with the old top five.

Then normalisation, measured the same way: documents matching revenue went from 221 to 278, up 26%; the vocabulary shrank from 37,933 to 23,676, down 37.6%. The value of the smaller vocabulary is not memory — it is that the document counts became correct, so inflated weights fell back to their real level.

Zero of five overlap is not a ranking perturbation. It means the results had been decided by filler words, and I would never have found that by looking at answers — only by reading the implementation and printing the per-term score contributions.

While shipping the tokeniser change I found two things that were not part of it. The keyword indexes could not be rebuilt in that working copy — no vector database and no API key present — and separately, two modules pointed at chroma_db/ for the vector store while the database that actually existed lived at data/chroma/. Two path conventions, disagreeing.

Decide what to do with a real problem discovered during unrelated work.

Neither of the two tempting options. Not fixing it inside an unrelated change, which makes the change unreviewable. Not staying silent, which is how a known problem becomes an unknown one.

Instead I recorded both in the project's debug log with a status — unresolved or known limitation — the evidence that located it, and what remains to be done. The project rule I hold myself to is that the record must contain the measured numbers and how I found them, not just the conclusion. "Changed K to 20" is worthless; "the weight of the was clamped to 1.811, above Group at 1.542" is the actual finding.

The path inconsistency is still there, and it is written down with the evidence rather than living in my memory. When someone hits it, the diagnosis is already done.

Two quite different paths reach the answer step. The retrieval path, which gathers documents, drafts, and grades. And a direct-answer path where the router decides no retrieval is needed and answers inline — a greeting, or "what does EBITDA stand for". The second path never touches retrieval, so none of the retrieval-side checks apply to it.

Guarantee that nothing empty or degenerate reaches the user, on any path including ones that do not exist yet.

Put one gate on every edge into the answer node rather than a check inside each branch. It is deterministic, makes no network call, and has exactly two rules: fail if the aggregated context is missing or empty, fail if it is shorter than a minimum length.

Failure routes to a fallback node that does not simply apologise — it distinguishes "nothing was retrieved" from "documents were retrieved but could not be synthesised", because those are different failures, and then lists the companies, financial years and metrics that are actually covered with two example questions.

A new branch into the answer node inherits the guarantee automatically — the property is structural rather than something a future contributor has to remember. Five unit tests pin the behaviour, including whitespace-only input and the trimming order.

In financial answers the failure that matters is a confident wrong number. A retrieval system has one advantage over a bare model here: I know exactly which documents the draft was written from, so there is something to check against.

Detect claims that are not supported by the retrieved evidence, and separately detect answers with no useful content — without one judgement contaminating the other.

One grading call with the question, the graded documents and the draft, doing two explicitly separated tasks and returning a strict structure — a pass-or-fail verdict plus a list of unsupported claims.

The strictnesses are deliberately opposite. The pass-or-fail task is instructed to be lenient — fail only when the answer is empty or is nothing but disclaimers — because that gate exists to catch "I don't have enough information", not to police quality. The groundedness task is strict: flag a claim only if its value is genuinely absent from every document.

The count then maps to a verdict: zero unsupported claims is yes, up to three is partial, more is no — and that becomes the confidence shown to the user, high, medium or low.

Confidence displayed to the user is derived from a check that actually ran, rather than from asking a model how sure it feels — which produces a number with nothing behind it. Medium and low confidence attach a caveat naming the specific claims; high confidence attaches nothing, because a caveat on every answer is noise.

Financial tables put the unit in a header cell and leave the data cells bare. So a chunk contains a header row saying US$M and a data row saying 29,016. The answer correctly says "US$29,016 million". The verifier saw two different strings and flagged a correct claim as unsupported.

This was not rare. It was systematic across every table-derived answer.

Make the verifier able to connect a bare number to its unit — without loosening it so far that it stops catching real invention.

Two layers. First, structural: for table-derived documents, scan the header lines for a unit token — US$M, A$B, A$'000, % and similar — and inject an explicit annotation above the content saying that all bare numbers in this table are in that unit. The reconnection happens before the model sees the document, rather than being left to inference.

Second, instructional: spell out in the prompt exactly which differences do not count as unsupported. Unit and scale formatting, minor rounding, accounting brackets for negatives, metric-name variants like underlying EBITDA supporting a claim about EBITDA, and "year ended 30 June 2024" meaning FY2024.

Correct answers stopped being flagged. That matters more than it sounds: a warning that fires on correct answers trains users to ignore warnings, which destroys the value of the one that matters.

When the verifier finds an unsupported claim, that claim gets named in the caveat shown to the user. The obvious implementation is to have the verifier return the claim it found — which means returning the sentence, including the figure.

Work out what the guard's own output does when it reaches a human.

Instructed the verifier to write a short label of at most eight words naming the metric and its context — "RIO FY2025 revenue", "BHP FY2024 EBITDA" — and explicitly not to reproduce the sentence or embed the value. The prompt carries both a good and a bad example so the constraint is unambiguous.

The reasoning: if the label contained the figure, then the message whose entire purpose is to flag a possibly invented number would print that number again, in the answer, as text. A user skimming would read the figure and miss the framing. The safety message would become another place the wrong number appears.

The caveat tells the user which claim to distrust without restating the value they should not rely on. The same labels are then reused as retry search terms, which is a second reason short and metric-shaped is the right format.

Comparison questions are the main use case — rank five companies by EBITDA, compare two on revenue growth. Retrieval fans out per company. If one company's documents come back empty, the model is holding four companies' figures and a question demanding five. The most likely completion is to quietly use a nearby number.

That is the worst failure this system has. A wrong figure attributed to the wrong listed company, stated fluently, with a citation next to it.

Make missing coverage impossible to paper over.

Three layers, because a prompt rule alone is not enforcement.

• Instruction: every excerpt header carries its company, and the synthesis prompt states that a citation may only support a claim about the company that document belongs to, and that a missing company must be reported explicitly as "data not found in provided documents" rather than estimated, inferred, or borrowed.
• Structure: a per-company found-or-not flag, initialised from the companies named in the question. It is then corrected after document grading — a company counts as found only if it has a document that survived grading, which is stricter than "retrieval returned something".
• Retry: if any company is still unfound, retrieval retries targeted at just those companies, reusing their company-specific query and appending the failed claim labels as extra search terms. Bounded at two attempts.

The per-company flags are written by parallel branches, so they need a merge rule that combines dictionaries rather than the default overwrite — otherwise the last branch to finish erases the others' findings.

Partial coverage becomes explicit in the answer instead of being silently filled in. The structural flag is the actual guard; the prompt rule is a first line of defence that I do not rely on.

The model is given a numbered source list and asked to place markers like [1] or [1,3] after each fact. Two things go wrong on their own. It sometimes cites a number outside the list. And it typically uses three of the eight sources it was offered, leaving five entries in the reference list that support nothing in the answer.

Guarantee that every marker shown to a user resolves to a real source, and that the reference list contains only what the answer used.

A fixed post-processing order, and the order is the whole design.

• Validate: scan the answer for markers, keep only numbers within the valid range, deduplicate, and preserve first-appearance order. An invented number is dropped, not rendered.
• Renumber: remap the surviving numbers to a contiguous sequence starting at 1, so the reader never sees a jump from [1] to [4].
• Filter: build the reference list from only the cited sources, and deduplicate on company, financial year and page, so two chunks from the same page do not become two entries.
• Then convert markers to superscript HTML with an index attribute the frontend uses for hover previews.

That last step must run last: superscripting first would break the pattern the earlier steps match on. It is written into the function's documentation because the ordering constraint is invisible from the call site.

No dangling citations and no unused references, so the citation list is a precise record of what the answer actually used — which is what makes a claim checkable against the source PDF.

Almost every check in the pipeline is a network call to a third party — two grading models, a reranking API, a routing model. Each one can time out, get rate limited, or return something unparseable. Left alone, each exception handler ends up making an availability decision by accident.

Decide, per layer, whether failure of the check should let the request through or block it — and have a stated reason, not a habit.

Wrote down a rule and applied it consistently. If the layer is a precision improvement — it makes a good answer better — it fails open, because losing it degrades quality while blocking on it removes the service. If the layer is a correctness floor — without it a wrong or empty answer reaches the user — it fails closed.

• Document relevance grader → open. On error, pass all retrieved documents through rather than none. A noisier context beats no context.
• Groundedness grader → open. On error, mark as passing. The one I am least comfortable with, and I say so: it is the hallucination check and its failure is silent.
• Reranker → open. On error, keep the fusion ordering. It is a precision pass over a ranking that already exists.
• Router → safe rather than open or closed. On error, default to retrieving and never to blocking on clarification. Retrieving unnecessarily costs money; refusing a valid question costs trust.
• Final content gate → closed, by construction. Deterministic, no network call, nothing to fail. That is why the floor is the dumb layer.

No single third-party failure can take down the answer path, and each degradation is a chosen one. It also produces a clean artefact to hand a risk function: a table of layer, failure mode, direction and blast radius.

"Add memory" was really four unrelated requirements: conversations must survive a restart, history must not grow past the model's limit, follow-up questions must keep their subject, and repeated questions should not cost full price.

Build each as its own layer with its own store and lifetime, and decide the order they run in.

• Durable state: the whole graph state is checkpointed to PostgreSQL after every step, keyed by thread. Survives restarts and makes a run replayable.
• Compression: past 10 messages, everything except the most recent 4 is summarised into one note and the originals are deleted. If summarisation fails, nothing is deleted — a failed compression must never lose history.
• Entity carry-over: company tickers and financial years are pulled from the last 6 messages and used to top up a question that is missing them.
• Answer cache: past question-and-answer pairs in their own vector collection, separate from the report chunks. Above 0.95 similarity, return the stored answer and skip retrieval; between 0.80 and 0.95, run the pipeline but seed it with the old answer.

The order is load-bearing: compress first so the entity scan reads a clean history, enrich second so the cache lookup uses the enriched question, cache last so a hit can skip everything downstream. Two separate vector collections means the system can never cite its own past answer as if it were an annual report.

Four mechanisms that can each be reasoned about, tested and turned off independently — rather than one "memory" component whose behaviour nobody can predict.

A single similarity threshold forces an unpleasant choice. Set it low and you get more reuse plus confident wrong answers — "BHP revenue FY2024" and "BHP revenue FY2023" are nearly identical as sentences and completely different as questions. Set it high and the cache almost never fires, so it is not worth having.

Get value out of near-matches without betting the answer on a similarity score.

Split the decision in two. Above 0.95, treat it as the same question and return the stored answer directly. Between 0.80 and 0.95, do not trust it as an answer — run the full pipeline as normal, but pass the old answer into synthesis labelled "past analysis on a similar question, for reference only", with the instruction to ground the answer in the retrieved excerpts regardless.

The middle tier changes the failure mode. A wrong near-match there does not produce a wrong answer; at worst it is misleading context that the retrieved evidence overrides.

The band where similarity scores are least reliable is exactly the band where a mistake is cheapest. Cached answers also carry their source list, so a cache hit shows the user the same citations as a live answer rather than looking like an unsourced claim.

The obvious instinct is to store the answer, because the answer is the valuable part. But at lookup time all I have is a question.

Pick the stored text so that the comparison being made is the comparison I actually want.

Store the question as the searchable content, and keep the answer, the companies, the financial year and the source list alongside it as metadata. So the search compares question against question.

Comparing a question against an answer is a mismatch of text shapes — a question is short and interrogative, an answer is long, full of figures and qualifications. They sit in different regions of the embedding space even when they belong together, so the scores get noisy and the threshold becomes meaningless.

The stored question is the enriched one, so a follow-up that resolved to "dividends [BHP, FY2024]" is later findable by someone who asks the full question directly.

Similarity scores that mean something, which is what makes a threshold a decision rather than a guess.

"What about their dividends?" has no company and no year in it. Sent to retrieval as written, it searches for the word "dividends" across five companies and returns noise.

Resolve the reference before retrieval, on the cheapest mechanism that covers the cases that actually occur.

Chose the narrow deterministic option over a model rewrite. In this domain the ambiguity is nearly always the same two slots — which company, which financial year — so I scan recent messages with two regular expressions plus a small full-name alias map, and append only what the current question is missing.

Three details that make it safe. It scans backwards so the most recent company wins when the user switches. It only fills a slot that is actually empty, and if the question already names both, it does not write to state at all. And the enriched question is what the cache lookup uses, so lookup and save share one key — otherwise you save under one text and search under another and never hit.

Zero added latency, zero added cost, fully deterministic and unit-testable, and it cannot invent a company that was never mentioned — which a model rewrite occasionally will.

A cache of the system's own answers has a failure mode a document cache does not: it can poison itself. One wrong answer written to the cache becomes a permanent wrong answer, served confidently to everyone who asks a similar question — and served faster than a correct one, because a cache hit skips retrieval entirely.

Worse, it is invisible. No error, no failed check, nothing in the logs. The system looks like it is working unusually well.

Decide what qualifies an answer to become memory — and it cannot simply be "we produced one".

Made the write conditional on the guardrail layer that already exists. An answer is saved only if the quality grader passed it and the groundedness verdict is not "no". A low-confidence answer still reaches the user — with its caveat — but it is never written to memory.

The second decision is the ordering, and it is the part worth bringing up. The save happens before the confidence caveat is appended, so what gets cached is the clean answer. Otherwise a future user asking a similar question would receive a stale warning about a verification run that had nothing to do with their question — a caveat outliving the doubt that produced it.

Stated as a principle: the read path is optimistic, the write path is pessimistic. A cache miss costs a few seconds and some credits. Poisoning the cache costs a wrong answer about a listed company's finances, repeatedly, until someone notices.

Memory quality is bounded by the same check that governs what the user is told to trust, so the two cannot drift apart. It is also the single place where all three of my main topics turn out to be one mechanism: the evaluator is the runtime guardrail, and the guardrail is what gates memory.

Both single-method options fail on this data. Vector search alone is weak on exact tokens — ticker codes, "FY2024", a specific figure — because meaning-based matching does not privilege an exact string. Keyword search alone misses paraphrase entirely: "profitability" does not match "EBITDA".

Combining them exposes a third problem. Fusing two ranked lists works on rank position and never reads the text, so a chunk containing the exact figure can sit below topically similar boilerplate that both methods liked.

Get both kinds of recall, then let something that actually reads the content decide what survives.

Three stages. Keyword and vector search each return 20 candidates, fused with weights 0.45 keyword and 0.55 vector — vector weighted slightly higher because paraphrase is the more common need, and the comment in the code says to raise the keyword weight if exact financial terms start to matter more.

Then a cross-encoder reads the question against each of the forty candidates and returns the best 8. Widening the pool raises the ceiling of what can be found; the reader decides the cut. Downstream token cost is unchanged because the final count is the same.

One implementation note worth having ready: a fresh reranker client is constructed per call, because the client holds an HTTP session that is not thread-safe and five company branches run in parallel. Construction costs microseconds against an API call of roughly 800 milliseconds.

Recall ceiling raised without raising the context cost, and the final selection made by content rather than by rank arithmetic. This is the change measured by the A/B harness in card 4 of the evaluation section.

With one global keyword index, a question about FMG spends most of its twenty candidate slots on whichever companies happen to score well — and BHP's reports are far longer, so they dominate. For a five-company comparison, the company with the thinnest reporting gets squeezed out entirely, which is exactly the guardrail failure in section 2, card 5.

Guarantee each requested company its own recall budget.

One keyword index file per company, built from only that company's chunks, plus a vector-store metadata filter restricting semantic search to the same company. A query-rewrite step decomposes the question into one company-specific query per company, and the graph fans out one retrieval branch per company to run in parallel.

Merging parallel branches needs explicit rules: retrieved documents accumulate by list concatenation, and the per-company found flags use a dictionary-merge rule so branches do not overwrite each other's keys. On retry, the fan-out naturally targets only the still-missing companies, because the rewrite step has already narrowed the query list.

Every named company gets its own twenty candidates and its own reranking pass, so a comparison across five companies has five independent recall budgets rather than one shared one. Wall-clock stays close to a single company's latency.

Text extraction from a PDF destroys tables. A financial table becomes a stream of numbers with the column headers, the units and the row labels all disconnected — and that is precisely the content the product exists to answer questions about.

A vision model reads the table as a table. But these are 300-page documents across fifteen report-years, and most pages are narrative text where a vision model adds nothing.

Find the pages worth paying a vision model for, then get structure — not just text — out of them.

Two stages, with a cheap deterministic filter in front of the expensive model.

• Filter: a PDF library detects tables, and I combine that with numeric density — digits divided by non-whitespace characters — requiring above 5% digits, at least 20 digit characters and at least 100 text characters. Vector drawing path counts catch charts. There is a specific guard for the common false positive: a detected "table" covering more than 80% of the page is usually a two-column text layout, so those are judged on absolute counts rather than density.
• Extract: a vision model with an explicit nested-cell JSON schema — headers, rows, column spans, and cells that split into sub-rows, to a fixed maximum depth — plus four few-shot page images with hand-written correct answers. It is instructed to output null for any cell it cannot read rather than guessing.

Operationally: parallel workers with checkpoint files so a run resumes rather than restarting, and embedding batched with pacing to stay under the API's per-minute token limit.

Tables survive ingest as structure, which is what later lets the groundedness check reconnect a bare "29,016" to its "US$M" header. The prompt also carries hard-won specifics — an "=" sign is never a cell separator, because remuneration tables kept getting split down the middle of a threshold condition.

Plenty of questions need arithmetic on retrieved figures — a growth rate, a margin, a three-year average. Language models make arithmetic errors, and a financial answer with a confidently wrong growth rate is worse than no answer, because it looks exactly as authoritative as a correct one.

Make the arithmetic deterministic and inspectable, and decide when it is needed.

An agent with typed tools rather than free-form calculation: growth rates and compound annual growth rate from a chronological series with matching period labels, ratios, averages, and a general expression evaluator for the steps that do not fit.

Two decisions inside that. The general evaluator uses a restricted expression library rather than the language's built-in evaluation, so a model-generated string cannot execute code — an obvious injection surface if you reach for the convenient function. And the tools validate their inputs and return an error string, so mismatched arrays or a division by zero come back as a message the agent can react to rather than an exception that kills the run.

The decision of whether arithmetic is needed is made after retrieval and aggregation, not from the question — so it is judged against the numbers actually available, rather than guessed from wording.

Every computed figure comes from real arithmetic, and the tool output shows its working — which makes a derived number auditable in the same way a retrieved number is citable.

The graph routes on model decisions — needs retrieval, needs news, needs clarification, out of scope. Those have to come back as reliable typed fields; a free-text answer that has to be parsed will eventually be parsed wrongly and route a request into the wrong branch.

The framework's default way of enforcing a schema is rejected by the model provider I use.

Get schema-enforced output from this provider, and make the constraint visible to whoever reads the code next.

Declared each decision as a typed schema with defaults and per-field descriptions — the descriptions are part of the prompt the model sees, so they do real work — and forced the structured-output method to function calling rather than the default. Temperature is zero everywhere, since none of these calls want creativity.

Two separate model clients exist for a specific reason: a small one capped at 1,000 output tokens for routing and grading, and a larger one at 5,000 for synthesis and final answers. A routing decision cannot accidentally be billed and latency-budgeted like a synthesis call.

The provider constraint is written as a one-line comment above each occurrence, because it is not discoverable from the code — someone "tidying up" the method argument would break routing in a way that only shows up at runtime.

Routing decisions arrive as typed booleans. Every one of these calls is also wrapped so that a failure falls back to a safe default rather than propagating.

When a question is too vague, the right behaviour is to ask. But asking means stopping mid-graph and waiting for a human, and the wait can be a minute or a week, across a server restart or a deployment. Holding it in memory means the reply arrives to nothing.

Pause execution durably, resume at the same point with the user's answer.

Used the framework's interrupt mechanism on top of a PostgreSQL checkpointer: the state is already persisted after every step, so the pause is just the absence of a next step. The API resumes by sending a resume command against the same thread, and the clarify node routes straight back into the routing decision with the enriched question.

The connection pool needed non-obvious settings — autocommit on, and the prepared-statement threshold set to zero, which is what makes it work behind a connection pooler. Progress is streamed to the frontend as server-sent events so the user sees which node is running rather than a spinner.

Conversations are resumable across restarts, and the same persistence gives replayability — I can reconstruct which route was taken, which documents survived grading, and what the graders said, which is as much an audit property as an operational one.