Intent recognition in Mines Financial

The short answer

Intent exists. It is just not stored as a label.

The closest thing to an intent classifier is one node, nodes/retrieve_decision.py. It makes a single LLM call and returns a structured object — but that object is not an intent name. It is four independent booleans:

class _RoutingDecision(BaseModel):
    is_out_of_scope: bool
    needs_retrieval: bool
    needs_news: bool
    needs_clarification: bool
    clarification_question: str
    direct_answer: str

Search the codebase for the word intent and you find nothing but prose. There is no intent field in either state schema. What exists instead is a flag vector, and that choice has a specific consequence worth naming:

Everything else in the pipeline is either a deterministic edge reading those flags, or a narrower decision made later once there is more evidence to decide with. The rest of this page is that sequence.

The shape of the system

Two graphs. A supervisor, and a retrieval subgraph mounted inside it as a single node.

SUPERVISOR  (state: MainState)

compress_context          ← entry point
   ↓
memory                    ← entities, follow-ups, cache
   ↓  (cache hit? → answer)
retrieve_decision         ← the four flags
   ↓
   ├─ out of scope        → answer (canned reply)
   ├─ vague               → clarify → back to retrieve_decision
   ├─ needs retrieval     → retrieval_agent  ⎫ in parallel
   ├─ needs news          → news_agent       ⎭
   └─ neither             → guardrails
   ↓
aggregate                 ← merge sources, decide on arithmetic
   ↓  (arithmetic? → calculator_agent)
guardrails                ← is there enough context at all
   ↓
answer  |  fallback

RETRIEVAL SUBGRAPH  (state: RetrievalState)

query_rewrite             ← split the query per company
   ↓
retrieve_company ×N       ← one per company, in parallel
   ↓
grade_docs                ← keep only relevant documents
   ↓  (nothing good? → retry, max 2)
synthesize                ← draft the answer
   ↓
grade_answer              ← quality + groundedness
   ↓  (unsupported claims? → retry, max 2)
END

Every LLM decision in both graphs uses the same mechanism — llm.with_structured_output(Model, method="function_calling") at temperature 0, against DeepSeek. The function_calling method is forced rather than chosen: a repeated comment in the source records that DeepSeek rejects the json_schema response format. Small detail, but a real one, and the kind of thing worth mentioning as evidence you built it rather than read about it.

Before the decision: compressing the history

nodes/compress_context.py — the graph's entry point.

Nothing about intent yet, but it shapes what the classifier will see. Once a conversation passes ten messages, the older ones are summarised and the four most recent kept verbatim:

COMPRESS_THRESHOLD = 10
KEEP_RECENT       = 4

What makes it relevant here is that the summarisation prompt is itself entity-directed — it is told to preserve exactly the things the router will later need:

Focus on: ASX mining companies discussed,
financial metrics mentioned (revenue, profit,
EBITDA, capex, dividends), fiscal years, and
any key conclusions reached.

If summarisation fails, the node returns compressed: False and leaves the messages intact rather than dropping them — the first of several fail-open choices in this codebase.

Slot extraction, done with regex on purpose

nodes/memory.py — companies and fiscal years, before any model sees the query.

The concepts page separates intent from slots. This project extracts its slots first, and deterministically:

_TICKER_RE = r"\b(BHP|RIO|FMG|MIN|NST)\b"
_FY_RE     = r"\bFY20\d{2}\b"

_ALIASES = {
    "fortescue":         "FMG",
    "rio tinto":         "RIO",
    "mineral resources": "MIN",
    "northern star":     "NST",
}

Two slots, both closed sets: five tickers, and fiscal years matching a fixed pattern, normalised so fy2024 becomes FY2024. No model call, no latency, no variance.

The honest caveat, which is better volunteered than discovered: this same pattern exists in three separate places — nodes/memory.py, utils/chart.py, and again inside the eval runner. Three independent copies of the same ticker regex is duplication waiting to drift, and the fix is one shared module.

Resolving follow-ups before classifying them

nodes/memory.py — _enrich_query. The step that makes conversation work.

Here is the problem. A user asks two questions in a row:

Turn 1:  "What was BHP's FY2024 revenue?"
Turn 2:  "What about their dividends?"

Turn 2 on its own is unroutable. There is no company in it and no year — the word "their" carries both, and it only means something given turn 1. Send that to the router as-is and the best possible outcome is a clarification question the user has already answered.

So before routing, the node scans back over the previous six messages, newest first, and appends only the slots that are missing from the current query:

Enriched:  "What about their dividends? [BHP, FY2024]"

Three details that make this cheap rather than clever: it exits immediately if both slots are already present in the query; it scans in reverse so the most recent mention wins when a conversation has moved from one company to another; and it appends a hint rather than rewriting the sentence, so nothing about the user's actual wording is lost.

The short-circuit that skips intent entirely

memory/semantic.py — two thresholds, three outcomes.

Before routing, the query is compared against previously answered questions by meaning rather than wording. One similarity score, two thresholds:

CACHE_THRESHOLD   = 0.95   # return the old answer
CONTEXT_THRESHOLD = 0.80   # seed the pipeline with it

The routing consequence is worth stating plainly, because it surprises people: on a cache hit the intent classifier never runs. The system has decided what you want without ever asking the question, on the grounds that it answered this before.

The middle tier is the more interesting design choice. A binary cache is either a hit or a miss; this one has a band where the memory is treated as a hint rather than an answer — which is how you get value from near-misses without ever serving a stale answer to a question that was not actually the same.

The de-facto intent classifier

nodes/retrieve_decision.py — one LLM call, four flags, and three deterministic guards around it.

This is the node to talk about when someone asks how intent recognition works here. It answers four questions at once:

Plus two free-text outputs: clarification_question, and direct_answer for queries the model can handle without any retrieval.

Inside the prompt, and the guards around it

The prompt is not the whole story. The deterministic code wrapped around it is what makes it safe.

The system prompt opens by naming the job exactly as the concepts page would recommend — a role, a domain, and a decision:

You are a routing agent for an ASX mining
financial report chatbot. Analyse the user's
question and decide how to handle it.

It then defines each flag with include and exclude cases, and closes with a few-shot block mapping seven example queries to their flag combinations. That structure — description, include, exclude, examples — is exactly the intent-definition shape from part one, applied to booleans instead of labels.

Three guards that do not trust the model

There is also a smaller guard: if the model asks for clarification but leaves the question blank, a hardcoded fallback is substituted — "Could you clarify which company (BHP, RIO, FMG, MIN, NST), fiscal year, and metric you are asking about?" Structured output guarantees a field exists; it does not guarantee it is useful.

The routing edge is pure code

graph.py — route_retrieve_decision. No model, no ambiguity.

The flags are read by an ordinary function. It is worth quoting in full because its brevity is the point:

if is_out_of_scope:            → "answer"
if needs_retrieval or needs_news:
    if needs_clarification:    → "clarify"
    → fan out to the agents
→ "guardrails"

Four lines of deterministic logic separating the probabilistic part from the consequences. The model decides what; the code decides what happens. That separation is what makes the behaviour testable — and there are 18 unit tests over the five routing functions in this graph, none of which call an LLM.

The precedence order is itself a design statement: out of scope beats everything, clarification beats retrieval, and anything not matching falls through to the guardrail rather than to the answer.

Clarification is a real interrupt, not a message

nodes/clarify.py — the graph stops, the user answers, the graph resumes.

When a query is too vague, the node calls LangGraph's interrupt, which suspends execution mid-graph. The API surfaces it to the frontend as an event, and the user's reply resumes the same run with Command(resume=...):

user_answer   = interrupt({"question": question})
merged_query  = f"{original_query} {user_answer}"
return {"query": merged_query,
        "needs_clarification": False}

The merged query then loops back to retrieve_decision and is classified again from scratch. This is the concepts page's clarification policy, implemented — with one difference worth noticing: the answer is appended to the original query rather than replacing it, so nothing the user said the first time is discarded.

Query decomposition: one question, several searches

retrieval_nodes/query_rewrite.py — where a multi-company question is split apart.

Inside the retrieval subgraph, a second LLM call re-reads the query and produces three things:

global_query:     str
companies:        list[str]
company_queries:  list[{company, query}]

Its instruction is emphatic about splitting: always create one item per company whenever multiple companies are mentioned, even if the same metric is requested for each. The worked example in the prompt:

Input:
  "What was BHP's FY2023 profit and
   RIO's FY2024 dividends?"

Output:
  companies = ["BHP", "RIO"]
  company_queries = [
    {"company": "BHP", "query": "BHP FY2023 profit"},
    {"company": "RIO", "query": "RIO FY2024 dividends"},
  ]

This is the second intent axis. The router decided which sources; this decides how many searches, against which company's documents. Each becomes a separate retrieval with a hard company filter applied twice — once in the vector store query, once again during deduplication.

The node also initialises a per-company completion tracker, company_status, which is what makes targeted retries possible in the next section.

Fan-out, and retries that narrow

agents/retrieval_agent.py — parallel dispatch and two capped retry loops.

The per-company queries are dispatched in parallel. If the query named no company at all — a general question — it broadcasts to all five instead:

if company_queries:
    one Send per company
else:
    Send to all of BHP RIO FMG MIN NST

Then two independent retry gates, both capped at two attempts:

The part worth showing off is what happens on the way back. query_rewrite on a retry does not call the LLM again. It reads company_status, works out which companies are still missing, and rebuilds queries only for those — with the failed claim labels appended as extra search terms:

missing = [c for c, found in company_status.items()
           if not found]
hints   = [h for h in unsupported_hints if c in h]
query   = f"{base} {' '.join(hints)}"

There is a neat coupling here that is easy to miss and good to point out: the grading prompt requires unsupported claims to be labelled in eight words or fewer, without the actual figure — "RIO FY2025 revenue", not the full sentence. That constraint exists because those labels are reused as search queries, and because a ticker is matched inside the label by substring to decide which company to re-fetch. A prompt formatting rule, load-bearing for the retry mechanism.

The decision made deliberately late

nodes/aggregate.py — "does this need arithmetic?", asked after retrieval rather than before.

Whether an answer needs calculation could obviously be decided up front, from the query — it looks like just another routing flag. It is decided afterwards instead, and the source says why in a comment: the decision is made now that we can see what data was actually retrieved, rather than guessing from the query alone.

The prompt reflects it — needs_calculation is true only if the context contains raw numbers that must be combined, and false if the context already contains a pre-computed answer.

This node also merges the parallel branches back together under fixed section headers — 【Annual Report Data】 and 【Recent News】 — so that downstream, the answer and the groundedness checker can tell which source a fact came from.

The one place a model picks a tool

agents/calculator_agent.py — everywhere else, "tool selection" is a graph edge.

Part one described the third implementation style, where intent is implicit in tool selection. This project uses it in exactly one place. The calculator is a ReAct agent holding four tools, and it chooses among them freely:

Its prompt carries the constraint that matters: never guess or estimate financial figures — only use numbers present in the context, and for growth rates, confirm chronological order.

The contrast with the news agent is the point. The news tool has a full selection docstring — "use for recent events… do NOT use for historical financial figures" — and no model ever reads it. It is invoked unconditionally, because the decision to use news was already made by needs_news back at the router. That docstring is dead metadata, and noticing it is the difference between describing your architecture and knowing it.

Confidence exists — but not where you'd expect

Not on the routing decision. On the answer's grounding.

Part one argued that a mature intent layer emits a label plus a confidence. This project emits no confidence on any routing decision. The single confidence value in the system is derived at the very end, and it comes from counting:

n = number of unsupported claims

grounded = "yes"      if n == 0
           "partial"  if n <= 3
           "no"       otherwise

confidence = {"yes": "high",
              "partial": "medium",
              "no": "low"}[grounded]

So "confidence" here means how much of this answer is traceable to a retrieved document — not how sure the router was about what you meant. When it lands at medium or low, the answer gets a caveat appended listing up to three unsupported claims, each tagged with its basis:

At low, a further line is added asking the user to cross-check against the original reports. And the memory write is gated on the same signals — an answer is only remembered if it passed grading and was not ungrounded, so the semantic cache cannot be poisoned with a bad answer that a later question would then inherit.

Multi-intent on two axes

And one axis it does not handle, which is the more useful thing to know.

What is not decomposed is the metric axis. "BHP revenue and FMG dividends" splits cleanly, because the split is by company. "BHP revenue and BHP capex" stays a single query, because there is only one company in it — and it works anyway, since both figures live in the same set of documents and one search retrieves both. It is a limitation that happens to be harmless in this domain, which is exactly how to describe it: know where the ceiling is, and know why you have not hit it.

Out of scope, in three layers

Part one insisted on an escape hatch. This has three, at different depths.

The gap in the middle is worth owning: an unsupported company or year has no explicit check. "What was Apple's FY2024 revenue?" is not caught by is_out_of_scope — it is a financial question, just about the wrong company — so it goes through retrieval, finds nothing, and is caught by the guardrail and fallback. The user gets a correct outcome via a longer and more expensive path than necessary. There is an eval case for exactly this, which is how it is known rather than assumed.

One default deserves a second look. The guardrail route reads guardrails_passed is False — an identity check, so a missing flag routes to answer, not to fallback. That is intentional and documented in the tests: absence means the context was never checked, as on the direct-answer path, so it should pass rather than block. A subtle fail-open, written down on purpose.

How the routing is actually evaluated

evals/ — a dedicated routing suite, a boundary suite, and unit tests underneath both.

Part one's four evaluation layers, mapped onto what exists here:

The routing suite runs the routing nodes directly rather than the whole graph — classify, then rewrite if retrieval was requested — which keeps it fast enough to run often. Cases run from routing_001_greeting through routing_008_clarification_needed, and two are worth naming in an interview:

• routing_007_report_plus_news — an explicit multi-intent case expecting needs_retrieval and needs_news both true, with the rationale written into the case: some questions need both historical report data and current external context.
• routing_008_clarification_needed — "Tell me about this company." expecting retrieval and clarification, which pins down the exact precedence the routing edge implements.

The boundary suite tests the edges that break things: an empty query, a vague company, an unsupported company (Apple), an out-of-range fiscal year (FY2030), a no-context guardrail trip, and a prompt injection case asserting an instruction override is ignored. It also checks negative signals — that a fallback answer contains no citation markers, and does use one of the expected phrases like not found or insufficient.

One more absence worth volunteering: is_out_of_scope is not asserted by any eval case. The most consequential flag — the one that decides whether the system engages at all — is the one with no direct test. It would be the first case I added.

Weak points, owned

Volunteering these reads as engineering judgement. Being caught out on them reads as the opposite.

How to say it out loud

Ninety seconds, if they ask how you handled intent recognition.

Then the three points that show depth beyond the happy path, in rough order of how well they land:

• Slots are regex, and that was a decision. Five tickers and three fiscal years is a closed vocabulary — a model would be slower, costlier, non-deterministic, and able to invent a sixth company. Use models where the vocabulary is open.
• One decision is made deliberately late. Whether arithmetic is needed is decided after retrieval, because only then can you see whether the report already states the growth rate. Route at the point of maximum evidence.
• Every LLM node fails open, and that is defensible only because nothing here writes. The worst case is a wasted retrieval. Add a tool that moves money and that default has to invert.

That closing move does the most work in the room: it shows you know the difference between what you built and what the same design would need in a bank, which is the actual subject of the interview.