Memory, Context Management, Orchestration

1 · Memory systems

Storage outside the model, with an explicit lifetime.

The concept · starting from the dumbest version

The naive solution to statelessness is obvious and it is where nearly every project starts: keep the entire conversation in a list, and send the whole list every time. That genuinely works, for a while. It is the right first version.

It breaks in three stages as the conversation grows.

• Cost, first. If turn twenty re-sends all nineteen previous turns, you are paying for the same text twenty times. Cost grows with the square of conversation length, not linearly.
• Quality, second — and this surprises people. Models attend most reliably to the start and end of their input. A crucial figure buried in the middle of a very long input is measurably less likely to be used than the same figure in a short one. So a longer context can produce a worse answer.
• Hard failure, last. Eventually the input exceeds the window and the call errors outright.

So a memory system is what you build once "send everything" stops working. And the central insight is that "memory" is not one thing. It is four different requirements that share a name, with different storage, different lifetimes, and different consequences when they go wrong. Designing them as one component is the most common mistake.

The four kinds · and what each one is for

In this project · four layers, in a fixed order

Every request passes through all four before any retrieval happens, and the order is load-bearing rather than incidental.

Why that order

Compress first so the entity scan reads a clean, bounded history rather than fifty messages. Enrich second so the cache lookup uses the completed question — looking up "what about their dividends?" would match nothing useful, whereas "dividends [BHP, FY2024]" might. Cache last so that a hit can skip everything downstream, which is the entire point of having it.

One detail worth understanding properly

The cache stores the question as the searchable text, and keeps the answer, the companies, the year and the source list alongside it as metadata. That sounds backwards — the answer is the valuable part — but at lookup time all you have is a question, so storing the answer as the search key would mean comparing a question against an answer.

Those are different shapes of text. A question is short and interrogative; an answer is long, full of figures and qualifications. Their embeddings sit in different regions even when they belong together, so similarity scores become noisy and any threshold you pick becomes meaningless. Storing the question makes it a question-to-question comparison, which is the comparison you actually want.

2 · Context management

Deciding what goes into one specific model call. Often called context engineering.

The concept · the part people assume is trivial

Suppose memory is solved: you have the conversation history, a set of retrieved documents, a cached answer, and a summary of earlier turns. Now you have to make one model call. What do you put in it?

The naive answer is all of it, and it is naive for a reason worth stating precisely. It is not just that you might exceed the limit. It is that irrelevant content actively degrades the answer. Give a model eight documents where two are relevant, and it has to work out which two. Give it the same two alone, and it does not. Extra context is not free even when it fits.

So context management is assembly, not concatenation. And the thing that makes it a real discipline is that it is a decision per call, not per application — different steps in the same pipeline want completely different context.

The five jobs · what context management actually consists of

In this project · nine calls, nine different contexts

This is the part of the project I would point at if someone asked what context engineering looks like in practice. A single request can involve up to nine model calls, and no two receive the same context. Each one gets the narrowest input that lets it do its job.

Three cross-cutting decisions first, because they explain the table below:

• Two model clients, deliberately. utils/llm.py is capped at 1,000 output tokens and used for routing, grading and rewriting. utils/llm_large.py allows 5,000 and is used only for synthesis and the final answer. A routing decision cannot accidentally be budgeted like a synthesis call. Both run at temperature 0.
• Different fidelity for different jobs. Relevance grading sees each document truncated to 1,200 characters — enough to judge topic, cheap across twenty candidates. Synthesis sees full content, because a figure could be anywhere in a table. Same documents, two fidelities, chosen per purpose.
• Structure is added, not hoped for. Retrieved documents are converted to plain dictionaries with company, year, page and source type promoted to top-level keys, so every step formats headers like [3] FMG FY2024, p.39 [table] consistently — and the model can attribute a claim to a company without inferring it.

Every model call, and what it is given

Isolation, concretely

Content of different trust levels is fenced with visible markers rather than merged. The merged context uses bracket headers — 【Annual Report Data】, 【Recent News】, 【Calculation Result】 — and the final prompt instructs the model to prefer report data and to note the discrepancy if the sections disagree. Cached answers carry "for reference only". Each document carries its own company header so a figure cannot drift between companies.

3 · Orchestration

Deciding which steps run, in what order, and who makes that decision.

The concept · two naive versions, failing in opposite directions

A question like "compare BHP and Rio on FY2024 revenue growth" cannot be one model call. It needs: work out what is being asked, turn it into searches, search for two companies, judge what came back, draft, verify, compute a growth rate, format with citations. Multi-step work needs something coordinating the steps — that is orchestration.

There are two obvious ways to do it, and they fail in opposite directions.

Real systems sit between these, and the choice is per component, not per application. That is the single most useful idea in this section. Steps whose order you know go in a fixed graph; steps whose order genuinely depends on intermediate results get a bounded loop.

The spectrum · three levels of who decides

The building blocks · vocabulary you need to read the code

• State. One typed object every node reads from and writes to. Nodes return only the fields they changed, and the framework merges them in — so a node cannot accidentally clobber a field it never touched.
• Node. One step. A plain function: state in, partial update out.
• Edge and conditional edge. A fixed connection, or a function that reads state and returns which node comes next.
• Reducer. The rule for combining writes to the same field from parallel branches. The default is overwrite — which silently loses data when branches run concurrently, so lists usually need to append and dictionaries to merge.
• Fan-out and fan-in. Split one request into parallel branches, then merge. The same shape as map-reduce.
• Subgraph. A graph used as a single node inside a bigger one, with its own private state — so its internals stay out of the parent.
• Checkpoint and interrupt. Persisting state after each step, which is what makes it possible to stop mid-run, wait for a human, and resume in place.
• Supervisor pattern. A top-level graph that dispatches to specialist sub-agents and merges their results, rather than one agent doing everything.

In this project · all three levels at once

The architecture is a supervisor graph of 11 nodes, containing a retrieval subgraph of 5 nodes, containing a ReAct agent for arithmetic. So all three points on the spectrum appear in one system, each chosen for a different reason.

The separation that makes it work

The routing model does not choose the next node. It returns four booleans — is_out_of_scope, needs_retrieval, needs_news, needs_clarification — into state. A plain function then reads those and decides the branch.

That separation is the whole design. The model contributes the judgement it is good at, which is understanding what a question is asking. The code keeps control of the flow, which means the routing is a function you can unit-test with a dictionary and no network — and there are tests in tests/test_graph_routing.py doing exactly that.

Parallel fan-out, and why it needs explicit merge rules

A comparison question is decomposed into one search per company, and the graph dispatches one retrieval branch per company to run concurrently. Two things then have to be stated explicitly, because the defaults are wrong for concurrency:

• Retrieved documents accumulate by list concatenation rather than overwrite, so five branches produce a combined pool instead of only the last one's results.
• The per-company found flags merge as dictionaries via a small custom reducer, so branches write their own key without erasing the others. With the default overwrite, the last branch to finish would delete every other company's status — a bug that would appear only under concurrency and only sometimes.

On retry, the fan-out naturally targets only the still-missing companies, because the rewrite step has already narrowed the list of queries.

Bounded loops

The retrieval subgraph can loop: if graded documents are unusable, or the draft fails, or two or more claims came back unverified, it goes back to query rewriting. Three constraints keep that safe. A counter in state caps it at 2 attempts. Every retry must change its input — the failed claim labels become additional search terms — because a retry with identical input only spends money. And exhausting the retries is a defined outcome routing to the fallback response, not an error.

Two smaller decisions that are easy to miss

• The subgraph declares an output schema, so only the finished retrieval_result crosses back into the parent state. Its dozen internal working fields — retry counts, ungraded candidates, per-company queries — stay private. That is state isolation as a design choice rather than a naming convention.
• The clarification pause is durable. The clarify node interrupts execution and returns the question to the caller; the API resumes with the user's reply against the same thread, and the node merges the reply into the original question before routing again. It works only because state was already checkpointed after every step — the pause is simply the absence of a next step. Without persistence, "ask the user something" is a dead end whenever the process restarts.

One request, end to end

The same question traced through all eleven steps, marking which of the three topics is doing the work at each point. This is the section to re-read if the three still feel blurred.

Glossary

Every term used above, in plain words.