Deep dive · evaluation only
Evaluating a Probabilistic System: Twelve Concepts in Four Groups of Three
Everything I would say about evaluation in an AI Scientist interview, grounded in a retrieval system I actually built — including the honest list of what that system still does not measure. Guardrails and runtime controls are covered on the companion guardrails page; this page is evaluation only.
Why evaluating this class of system is hard
Read this first. Every one of the twelve concepts exists because of one of the three problems below.
Problem one: the model is not deterministic. Ask a large language model the same question twice and you can get two different answers. Both may be correct. Both may be wrong in different ways. Ordinary software testing assumes that the same input gives the same output, so a single test run tells you the truth. Here a single run tells you almost nothing. One good answer is not evidence that the system is good; one bad answer is not evidence that it is broken.
Problem two: often there is no single correct answer. If a customer asks "why is my card declining", there are many good replies and many bad ones, and no exact string to compare against. Some questions do have exact answers — "what was my balance on 3 March" — and those are the easy ones. The hard part of evaluation is designing a check for the questions where correctness is a range, not a value.
Problem three: the system has many stages, so one score cannot locate the fault. A modern assistant is a pipeline. It classifies what the customer wants, it rewrites the question, it searches documents, it filters what it found, it drafts an answer, it checks that answer, it decides whether to refuse. If you only measure "was the final answer good", then when the score drops from 82% to 71% you have learned that something broke and nothing about what. My own system has an eleven-node supervisor graph with a five-node retrieval subgraph inside it. A single end-to-end number would have been useless to me.
| Group | The question it answers | The three concepts |
|---|---|---|
| A | What do you measure? | Ground truth · component vs end-to-end · per-stage metrics |
| B | How do you measure it? | Deterministic vs judge · judge validation · rubrics |
| C | When do you measure it? | Offline · CI gating and cost tiers · online |
| D | How do you know it is real? | Variance · sample size · drift and baselines |
Why "four groups of three" — the memory trick, and why it matters under pressure
Twelve separate facts are hard to recall while someone is looking at you. Four labels with three items under each is much easier, because you only have to hold four things and then unpack one at a time. If the interviewer asks a broad question — "how would you evaluate this?" — the four group names are the answer, and the twelve concepts are the follow-up detail.
The order is also a real argument, not just a mnemonic. You cannot choose a method (group B) before you know what you are measuring (group A). You cannot decide where a check belongs in the release process (group C) before you know whether it is deterministic. And none of the first nine matter if you cannot tell a real improvement from noise (group D). Said in that order, the structure sounds like engineering judgement rather than a list.
How to use this page
Each of the twelve cards has exactly three parts, always in the same order.
- 1 · The concept — the problem first, then the naive approach and why it fails, then the real answer, then a concrete example set in a hypothetical NAB-style assistant.
- 2 · If this is missing — the terracotta banner. One specific failure with a plausible bank consequence. This is the part interviewers remember.
- 3 · STAR for the interview — Situation, Task, Action, Result, plus the bullet list to memorise and the literal questions this answers.
A · What you measure
Concepts 1–3Before any tooling: decide what "correct" means, decide at which level you will check it, and choose one number per stage.
Ground truth and the golden set
"How do you know what the right answer is?"
1 · The concept
The problem. You want to say "the system got 8 out of 10 right". To say that, you need to already know the right answer for each of the ten. Where does that knowledge come from? Nowhere, unless someone writes it down first. Ground truth is that written-down correct answer, and a golden set is the fixed, version-controlled collection of those input-and-answer pairs.
The naive approach, and why it fails. The naive approach is to run the system, read the answers, and judge them as you go. This fails for three reasons. First, your judgement drifts — what you accepted on Monday you reject on Friday, so yesterday's score and today's score are not comparable. Second, you unconsciously grade generously on the change you just made, because you want it to work. Third, nothing is reusable: next month you cannot re-run last month's judgement, so you can never prove you have not gone backwards.
The real answer. Write the expectation down before you run anything, store it in the repository next to the code, and make the check mechanical. Crucially, ground truth does not have to be the whole answer text. It is usually a much smaller claim that is easy to verify: a figure that must appear, a route that must be chosen, a tool that must be called, a refusal that must happen. Writing full model answers is slow and brittle; writing the one thing that must be true is fast and durable.
A NAB-style example. Imagine NAB's retail assistant. A golden set of 200 cases might contain: 40 fee questions where ground truth is the exact fee figure from the published fee schedule; 30 transaction-dispute cases where ground truth is "the dispute tool was called with the right transaction identifier"; 25 home-loan questions where ground truth is "general information given, no personal recommendation, and the words 'you should' do not appear"; 20 hardship cases where ground truth is "the hardship support path was offered and no automated refusal occurred"; and 15 out-of-scope cases where ground truth is "declined and offered a human". Notice that only the first 40 have a single correct string. The rest have a correct property. Both are ground truth.
Where golden sets come from — four sources, ranked by how much I trust them
- Real user traffic, labelled by humans. The best source, because it contains the questions people actually ask, including the badly phrased ones. Expensive, and in a bank it needs privacy handling before anyone can look at it.
- Authoritative documents. A fee schedule, a product disclosure statement, a filed annual report. Ground truth is copied out of the source, so there is no argument about whether it is right.
- Incident and complaint history. Every case that once caused harm becomes a permanent test. This is how a suite stops repeating its own history.
- Hand-written by the engineer. What my project has. Fastest to produce and honest about the pipeline's shape, but it only contains failures I personally thought of, which is the limitation I state openly in concept 7.
2 · If this is missing
Suppose no case in the suite covers offset accounts. The assistant tells a customer that their offset account reduces the interest on their loan — for a product variant that has no offset feature. The answer sounds fluent and authoritative, so nobody flags it. Six weeks later a complaint arrives, and because the same wording was generated for anyone who asked a similar question, the bank has to work out how many customers received it and what to do about them.
The cost is not the wrong sentence. The cost is the six weeks, the remediation exercise, and the fact that when the regulator asks "how did you test for this", the honest answer is "we did not, because we never wrote down what correct looked like".
3 · STAR for the interview
I built a retrieval chatbot over the annual reports of five ASX-listed mining companies, covering three financial years. Every answer is full of figures. I was changing prompts and retrieval settings, and my only feedback was reading a few answers and forming an impression.
Turn "this feels better" into something written down, so that two runs a month apart could be compared.
I wrote cases as JSON lines files, one case per line, in four suites: routing,
retrieval, calculation and boundary — 8, 8, 6 and 6 cases, so 28 in total. Every
case must have five fields: an id, a type, the query, an expected object, and
why_it_matters. The loader raises an error if any field is missing,
so a lazily written case fails loudly instead of silently passing.
Separately I wrote 12 numeric cases where ground truth is an exact figure string I had verified was present in the ingested text — for example 55,658 for BHP's FY2024 revenue and 29,016 for its FY2024 underlying EBITDA. Those are used only by the retrieval A/B harness.
40 case lines in total that I can re-run at any time. The
why_it_matters field turned out to be the most valuable one: it
forced me to justify each case, and when a case failed it told me immediately
whether I should care.
The honest limitation: I wrote every case myself, so the suite reflects my imagination rather than real user traffic. In a bank I would seed it from real transcripts and complaint history instead.
- Ground truth written before the run
- Five required fields, loader raises on missing
- 28 cases in 4 suites, plus 12 numeric
- Verified figure strings, not model answers
- Gap: hand-written, not real traffic
- "How did you know your changes were improvements?"
- "Where does your evaluation data come from?"
- "How do you define correctness when there is no single right answer?"
Component-level versus end-to-end evaluation
"Your score dropped five points. Which part broke?"
1 · The concept
The problem. The pipeline has many stages. A wrong final answer could come from any of them, and the stages interact: bad retrieval can still produce a good-looking answer, and good retrieval can still be ruined by a bad summary. You need to know which stage to fix.
The naive approach, and why it fails. The naive approach is end-to-end only: send the question in the front, grade what comes out the back, one number. It is intuitive and it is what the user experiences, so it feels like the right thing to measure. It fails in two specific ways.
First, it cannot localise. The score moves and you have no idea why, so you start guessing and changing things, which is the slowest possible debugging loop. Second, errors cancel out. Retrieval gets worse, the answer model compensates by leaning on what it already knows from training, the end-to-end score stays flat — and you have silently traded a grounded answer for a plausible guess without any measurement noticing.
The real answer. Keep both, and know what each is for. End-to-end evaluation is your measure of user-visible quality and your release signal. Component-level evaluation is your debugging and experimentation instrument. Component evaluation is also dramatically cheaper and less noisy, because when you test the retriever alone there is no language model in the loop at all — so the number you get back is exactly repeatable.
A NAB-style example. Imagine the assistant answers a fee question wrongly. End-to-end tells you the answer was wrong. Component evaluation tells you which of four things happened: the intent classifier sent a fee question down the transactions path; the document search returned the business fee schedule instead of the personal one; the relevance filter threw away the correct page; or everything upstream was right and the answer-writing step misread the table. Those are four completely different fixes. Without component evaluation you cannot even tell which conversation to have.
End-to-end
Measures what the customer gets. Slow, expensive, noisy — every stage's randomness stacks up. Right for release gates and for tracking overall quality over months. Wrong for debugging: it tells you that, never which.
Component-level
Measures one stage against a controlled input. Fast, cheap, often fully deterministic. Right for experiments and root-cause work. Wrong as your only signal: every component can improve while the assembled system gets worse.
2 · If this is missing
With end-to-end scoring only, someone upgrades the embedding model used for document search. Retrieval quality quietly falls, but the answer model is strong enough to produce confident replies from its own general knowledge of banking. The end-to-end score barely moves, so the change ships.
What has actually happened is that the assistant has stopped answering from NAB's own documents and started answering from what a general model believes about banks. Fee figures are now approximately right and specifically wrong. Nothing alerts, because the only instrument in place cannot see inside the pipeline — and when an auditor later asks which document supported a given answer, there is no honest answer to give.
3 · STAR for the interview
I wanted to test whether a bigger candidate pool plus a reranking step actually improved retrieval. Measured end-to-end, the difference was invisible: the answers varied run to run, and the answer model papered over gaps.
Measure the retrieval change on its own, with no language model anywhere in the loop, so the only thing that varied was retrieval configuration.
I wrote a second harness, evals/ab_retrieval.py, that calls
retrieve_company_node directly instead of running the graph. It scores
whether a document containing the known figure string reached the caller, and at
which one-based rank. To build the baseline I swapped the reranker for a
pass-through function, so the two configurations differed only in pool size and
reranking.
The main harness does the same thing at a smaller scale: routing cases call
retrieve_decision_node and query_rewrite_node directly,
and retrieval cases invoke only the retrieval subgraph — never the full
eleven-node graph.
A per-case verdict I could read at a glance: same,
NEW FOUND IT, NEW LOST IT, or
rank 6 -> 2 better. That is a component-level measurement with no
language model variance in it, so a rank change of one position is a real change
and not noise.
One implementation detail that mattered: the candidate pool size is baked into
the search index when it loads from its cached file, so I had to clear
_bm25_cache between configurations or the second run silently reused
the first run's setting.
- End-to-end = release signal; component = debugging
- Errors cancel out end-to-end
- No LLM in the loop = repeatable number
- Pass-through function makes the baseline
- Cache cleared between configs, or silent reuse
- "How do you debug a multi-stage pipeline?"
- "How do you isolate the effect of one change?"
- "Why not just measure the final answer quality?"
Per-stage metrics — one number per stage, and both error directions
"What specifically do you measure at each step?"
1 · The concept
The problem. Once you accept that each stage needs its own measurement, you have to choose what that measurement is. Choose badly and you get a number that always looks fine, or a number that moves for reasons you do not care about.
The naive approach, and why it fails. The naive approach is accuracy everywhere — "percentage correct" for each stage. It fails because accuracy hides the direction of the error, and in a bank the two directions have completely different costs. A refusal metric of 95% accuracy could mean the assistant refuses slightly too often, which is annoying, or that it answers questions it should have refused, which is a licensing problem. Same number, opposite severity.
The real answer. One primary metric per stage, chosen to match that stage's job, plus both error directions reported separately whenever a stage makes a yes-or-no decision. The five that matter for an assistant like this:
| Stage | Metric | What it is, in plain words | Why this one |
|---|---|---|---|
| Routing | Routing accuracy | Of the questions asked, the share sent down the correct path. | A routing mistake makes every later stage irrelevant. It is also cheap to measure, because the correct route is a single label. |
| Retrieval | Recall@k | Of the cases where a correct document exists, the share where at least one correct document appears in the top k results. | The answer stage can ignore an irrelevant document, but it cannot invent a document that was never fetched. Missing evidence is unrecoverable; extra evidence is usually survivable. |
| Answering | Groundedness rate | The share of answers in which every factual claim can be traced to a retrieved document. | This is the metric that catches confident invention, which is the failure mode that actually harms customers. |
| Refusal | Refusal correctness, both directions | Reported as two numbers: wrong refusals of things it should have answered, and wrong answers to things it should have refused. | The two costs are not comparable, so a single blended number destroys the information you most need. |
| Whole system | Latency and cost per query | Wall-clock time to reply, and money spent per reply. | Quality improvements are usually bought with time and money. Without these two you will "improve" the system into something nobody can afford to run. |
A NAB-style example. A hardship conversation shows why both refusal directions must be separate. A customer writes "I have lost my job and cannot make this month's repayment". If the assistant wrongly refuses — treats this as financial advice and declines — a person in genuine distress is turned away by an automated system at the worst possible moment. That is a real harm and a vulnerable-customer failure, not merely a poor answer. If instead the assistant wrongly answers a different question — "should I fix my mortgage rate?" — with a recommendation, it has provided personal financial advice, which is licensed activity in Australia. Both are refusal errors. One harms the customer, the other creates a regulatory breach. A single accuracy figure of 95% could be either.
Recall versus precision — and the two terms behind them, in plain words
Recall answers: of all the things I should have found, what fraction did I find? Precision answers: of all the things I did return, what fraction were actually right? They trade off. Return more documents and recall rises while precision falls.
Recall@k is recall measured only within the first k results, where k is however many you actually pass on. Mine is 8, so recall@8 is the share of cases where a correct document was among the eight the retriever handed to the next stage. Anything ranked ninth might as well not exist.
For retrieval feeding a language model, recall is the metric to optimise first. A modern model tolerates a few irrelevant documents in its context reasonably well. It cannot recover from the correct document being absent — it will either refuse or, worse, fill the gap from general knowledge.
2 · If this is missing
With only a blended quality score, the assistant slowly becomes more cautious — each prompt revision adds a little more hedging. Refusals on hardship questions creep from 3% to 19% over two months. The blended score improves, because refusals are never wrong in an obvious way and never generate an angry follow-up.
The visible outcome is that roughly one in five customers asking for hardship support is turned away by an automated system and does not try again. Nothing alerted, because the one metric being watched went up. That is the specific case where a wrong refusal is a real harm rather than an inconvenience, and it is invisible unless refusal errors are counted in both directions separately.
3 · STAR for the interview
My pipeline has distinct stages — a routing decision, a query rewrite, parallel retrieval per company, a relevance filter, answer drafting, and an answer grade. Each can fail in its own way.
Give each stage a metric that would actually move when that stage broke, rather than one overall score.
Routing accuracy comes from the routing suite: the case declares the expected
route and tool, and the check is an exact comparison. Retrieval gets a real
recall-style metric in the A/B harness: hit@top_k,
hit@5 and mean_rank over the 12 numeric cases.
Groundedness runs at runtime, not just in the suite — my
grade_answer node lists claims that no retrieved document supports,
and maps the count to yes at zero, partial at three or
fewer, and no above three, which becomes a confidence level of high,
medium or low on the answer.
Boundary cases cover the refusal direction where the system should decline or state that data is missing. What I do not have is systematic measurement of the opposite direction — over-refusal — and I do not record latency or cost per query at all.
When something regressed I could usually name the stage from the output alone,
because the failing check printed the specific mismatch rather than just
FAIL.
Two honest gaps. First, no latency or cost measurement, which I would add first in a production setting because it is nearly free to collect and it is what makes a quality-versus-cost trade-off arguable. Second, the page filter in my vision-ingest step — a heuristic based on how dense a page is with numbers — has never had its recall measured. It is the one stage in the pipeline with no metric at all, and it can silently drop a page containing the answer.
- One metric per stage, matched to its job
- Recall first for retrieval — missing evidence is fatal
- Refusal errors: two numbers, never one
- Latency and cost belong beside quality
- Gap: over-refusal, cost, and page-filter recall
- "What metrics would you track for a retrieval-augmented assistant?"
- "Why recall rather than precision for retrieval?"
- "How do you evaluate whether a system refuses correctly?"
B · How you measure
Concepts 4–6Pick the cheapest instrument that can settle the question. When the instrument is itself a model, treat it as an instrument that needs calibrating.
Deterministic checks versus LLM-as-judge
"How do you grade an answer automatically?"
1 · The concept
The problem. You have a golden set, and now you need something to decide pass or fail for each case without a human reading it. Some cases are easy — did the number 55,658 appear? Some are not — was this hardship reply appropriately empathetic and free of advice?
The naive approach, and why it fails. There are two naive approaches and both fail, in opposite directions.
The first is exact string matching for everything. It fails because the model rephrases. "Revenue was US$55,658 million" and "revenue of approximately 55.7 billion US dollars" both convey the truth, and a naive string match fails the second one. You then spend your life fixing false alarms instead of finding real bugs, and eventually you stop trusting the suite.
The second is asking a model to grade everything. This is seductive because it handles all cases with one implementation. It fails because a model grader is slow, costs money per case, is itself non-deterministic — so the same answer can pass on one run and fail on the next — and it is capable of getting simple factual comparisons wrong. Using a probabilistic instrument to check whether one number equals another number is strictly worse than using the equals sign.
The real answer, as a rule. Never send to a model what a regular expression or a numeric comparison can settle. Sort every check into three tiers and only escalate when the tier below genuinely cannot do the job.
| Tier | Instrument | Use it for | Properties |
|---|---|---|---|
| 1 | Exact or structural comparison | Route labels, tool names, whether a tool was called, whether a required field is present, whether a citation marker exists, exit codes. | Free, instant, perfectly repeatable. Cannot be wrong about what it checks. |
| 2 | Pattern and numeric checks | Is a figure present within a tolerance; does the answer contain a forbidden phrase; is a date in range; does the reply contain a required disclaimer. | Still free and repeatable. Needs care about units and formatting, which is real work but bounded work. |
| 3 | Model as judge | Is this claim supported by these documents; is this tone appropriate for a distressed customer; is this general information or personal advice. | Costs money and time, non-deterministic, needs its own validation. Only justified where meaning must be understood. |
A NAB-style example. For a fee question, tier 1 checks that the fee-lookup tool was called; tier 2 checks that the exact fee figure from the schedule appears in the reply and that the phrase "you should" does not; tier 3 is only needed to answer "did this reply cross from general information into personal advice", which genuinely requires reading and understanding the sentence. Three of the four checks on that one case are free and exactly repeatable. Only the fourth needs a model.
What "LLM-as-judge" means — and the two shapes it comes in
LLM-as-judge means using a language model to grade another model's output. You give the judge the question, the answer, usually the source documents, and a set of rules, and it returns a verdict.
- Pointwise. The judge sees one answer and scores it against a rubric. Simple, and what my project uses. Scores tend to cluster and drift, so absolute values are not very meaningful across time.
- Pairwise. The judge sees two answers to the same question and says which is better. More reliable than pointwise scoring, because comparison is an easier judgement than absolute rating — but it suffers from position bias, which is concept 5.
A useful middle option that is often forgotten: use a model to extract rather than to judge. Ask it to pull out the numeric claims it made, then compare those numbers deterministically. You get the model's language understanding for the hard part and an exact comparison for the part where exactness matters.
2 · If this is missing
Suppose every check in a 200-case bank suite is a model judgement. One full run costs several dollars and takes twenty minutes, so nobody runs it on a pull request — it becomes a weekly ritual, then a monthly one. Worse, because the judge is non-deterministic, the suite reports 186, then 191, then 184 out of 200 with no code change at all. Engineers learn that a five-point move means nothing, so when a real regression of five points arrives it is dismissed as noise.
The failure is not a wrong answer reaching a customer. It is that the evaluation system has stopped functioning as evidence while continuing to produce numbers, and a release gate built on it cannot honestly block anything.
3 · STAR for the interview
My checks split naturally into two kinds. Routing and calculation cases have exact expectations — a boolean flag, a company list, a tool name. Answer quality and groundedness need judgement.
Use a model only where a model was genuinely required, and keep everything else free and repeatable.
The offline suite is almost entirely deterministic. Shared helpers do exact comparisons: a boolean field check, a list-membership check that upper-cases both sides so a ticker matches regardless of case, and a missing-values helper that reports precisely which expected values were absent. Fiscal years are pulled from the query with a regular expression, and metric keywords with a case-insensitive substring scan against a fixed keyword list. No model is involved in any of that.
My production guardrail check is the same principle taken to its extreme: it is
a ten-character minimum-length test on the assembled context, with no model call
at all. A model would have added cost and latency to answer a question that
len() answers perfectly.
The two places I do use a model are both runtime graders where meaning has to be understood: a relevance filter over retrieved chunks, and a claim-level groundedness check on the drafted answer.
The offline suite runs at zero cost and gives the same result every time, so a failure is always a real failure. That is what makes it usable as a gate — a point I come back to in concept 8.
One refinement worth mentioning: the groundedness judge kept flagging correct figures in financial tables, because the unit lives in a header cell far from the number. I fixed that with a regular expression that finds the unit token in the header line and injects an explicit annotation — for example that all bare numbers in this table are in US dollars, millions. A deterministic pre-processing step made the model judgement reliable, rather than replacing the model.
- Rule: never ask a model what
==can settle - Three tiers: exact, pattern, judge
- Judge costs money, time, and repeatability
- My guardrail: a 10-character check, no LLM
- Fix the judge's input before blaming the judge
- "When would you use an LLM as a judge, and when would you not?"
- "How do you keep evaluation cheap enough to run often?"
- "How do you handle answers that are correct but worded differently?"
Judge validation — and the biases a judge arrives with
"How do you know your judge is any good?"
1 · The concept
The problem. The moment you use a model as a judge, that model is your measuring instrument. Every conclusion you draw about the system is filtered through it. An uncalibrated instrument does not give you a slightly wrong answer; it gives you an unknown answer that looks precise.
The naive approach, and why it fails. The naive approach is to write a good prompt for the judge, read a handful of its verdicts, agree with them, and trust it from then on. It fails because you checked the cases where the judge and you were both looking at the obvious, and you never measured the rate at which you disagree. Without that rate you cannot answer the only question that matters: if the judge says quality went from 78% to 84%, how much of that six-point move is real?
The real answer. Treat the judge as a model that itself needs an evaluation set. Take a sample of cases — a couple of hundred is a reasonable starting point — have humans label them, run the judge on the same cases, and measure agreement. Report raw agreement and a chance-corrected figure. Then read the disagreements, because they tell you which way the judge is wrong and let you fix the rubric.
Also expect three specific biases, which are documented properties of model judges rather than surprises:
- Length preference. Judges systematically prefer longer answers. A change that only made replies more verbose will register as a quality improvement.
- Self-style preference. A judge tends to prefer text that looks like its own output — same model family, same phrasing habits. If the judge and the system under test are the same model, the system is partly grading its own homework.
- Position bias. In a pairwise comparison the judge favours one slot, usually the first. The fix is cheap: run every pair in both orders and only count the comparison when the two runs agree.
A NAB-style example. Imagine the judge's job is to decide whether a reply crossed from general information into personal financial advice — a licensing boundary in Australia. Two teams compare prompt versions and the judge says version B is better. If nobody has measured judge agreement, the bank does not know whether the judge understands the advice boundary at all. Suppose validation later shows 71% raw agreement with a compliance reviewer, with the disagreements clustered on exactly the borderline cases the boundary is about. The judge is close to useless precisely where it is needed most — and every decision made using it needs revisiting. That is a discoverable fact that costs one afternoon of labelling.
Reading an agreement number — rough bands, and why chance correction matters
There are no universal thresholds, but rough working bands help. A chance-corrected agreement above about 0.8 means the judge can substitute for a human on that task. Around 0.6 to 0.8 means it is usable for tracking direction over time but not for absolute claims. Below about 0.5 the judge should not be used for that task at all, and the honest move is to narrow the question until it can.
Chance correction matters because of imbalance. If 90% of answers are genuinely grounded, a judge that says "grounded" every single time scores 90% raw agreement and has learned nothing. The chance-corrected figure for that judge is zero, which is the correct verdict.
One more comparison worth having ready: measure human-to-human agreement too. If two compliance reviewers only agree with each other 80% of the time, a judge at 78% is close to the ceiling for that task, and the right response is to sharpen the definition rather than to keep tuning the judge.
2 · If this is missing
An unvalidated judge with a length preference is quietly steering the product. Six weeks of prompt iteration, each round guided by the judge's score, produces replies that grow from 90 words to 260 words. The judge's quality score rises the whole time. Then real signals move the other way: customers abandon the chat more often, and contact-centre calls about the same topics go up, because nobody reads 260 words on a phone.
Six weeks of engineering effort was spent optimising a bias in the measuring instrument. The scores were real numbers, carefully tracked, and pointed the wrong way the entire time. The only way to have caught it was to measure the judge against humans before trusting it — which is exactly the thing my own project has never done.
3 · STAR for the interview
My project has two model-based graders running at runtime. One filters retrieved chunks for relevance. The other grades the drafted answer and lists claims that no source document supports, classifying each as arithmetic, an extrapolation from a trend, or general knowledge.
Be honest about what those graders are: measurement instruments I have never calibrated.
This is the biggest gap in my project. I have never validated either judge against human labels. There is no measured agreement rate, so when the groundedness grader says an answer is fully grounded, I have no quantified basis for believing it. I tuned both prompts by reading outputs and forming an impression — the exact naive approach I have just argued against.
What I would build first, and it is a day of work: sample 100 answer-and-document pairs from real traces, label groundedness myself twice a week apart to get my own self-agreement as a ceiling, run the grader over the same 100, and report raw agreement plus a chance-corrected figure and a two-by-two table of the disagreement directions. I would also check the length bias specifically, by testing whether the grader's verdict changes when the same claims are restated more verbosely.
Today the result is a stated limitation rather than a number, and I would rather say that clearly than imply a rigour I do not have.
One mitigation I did build deliberately: both graders fail open. If the grader call throws, the relevance filter passes all documents through and the answer grader records a pass. That keeps a grader outage from breaking the product, and it also means grader failures are invisible in the score — which is precisely why the agreement measurement matters. A fail-open instrument that is silently broken reads as "everything is fine".
- The judge is the instrument — calibrate it
- Agreement with humans, chance-corrected
- Three biases: length, self-style, position
- Both orders in pairwise, count only when they agree
- My gap: never validated; both graders fail open
- "How do you know your judge is any good?"
- "What are the known failure modes of LLM-as-judge?"
- "What is the weakest part of your evaluation setup?"
Rubrics and structured verdicts — constrain the shape, split the judgements
"How do you stop a judge from being vague?"
1 · The concept
The problem. A judge returns free text. You have to turn that into a number you can compare across runs, and you have to know what the number means. Free text does neither.
The naive approach, and why it fails. The naive approach is "rate this answer from 1 to 10". Three things go wrong.
First, the scale has no shared meaning — nothing distinguishes a 6 from a 7, so the same answer scores differently on different days and the score drifts as the judge model is updated underneath you. Second, the number is a blend: an answer that is factually wrong but beautifully written might score 7, and so might one that is factually perfect but rude. You cannot recover which. Third, parsing free text is brittle — the judge writes "I'd say around 8, though..." and your regular expression extracts something wrong.
The real answer, in two halves.
Constrain the output shape. Require the judge to return a structured object with named fields and a fixed set of allowed values, enforced by a schema rather than by asking politely. Then there is no parsing step and no ambiguity about what came back.
Split the judgements. Ask separate questions and get separate verdicts. Is it factually supported. Is it complete. Is the tone right. Did it stay on the correct side of the advice boundary. Each with a short, explicit rubric naming the exact conditions. Aggregate afterwards if you want a headline number — but keep the components, because the components are what tell you what to fix.
A NAB-style example. A hardship reply needs four independent judgements: was the factual content about hardship options correct; was the required support pathway offered; was the tone appropriate for a distressed customer; and did it avoid personal financial advice. Blended into one 1-to-10 score, a reply that is warm, accurate, helpful and contains one recommendation might score 8 — and that single recommendation is the licensing problem. Split into four verdicts, three come back pass and the advice check comes back fail, which is both the correct assessment and directly actionable.
Blended score
"Rate 1–10." One number, no shared meaning, drifts over time, hides which
dimension moved. A regression looks like 7.8 → 7.4, which is
unactionable and easy to dismiss as noise.
Split structured verdicts
Four named fields, each with allowed values and a written rubric. A regression
looks like advice_boundary: pass → fail on 6 cases, which names the
problem, the size, and the owner.
2 · If this is missing
A single blended quality score sits at about 7.9 for months. Underneath it, two things move in opposite directions: factual accuracy on fee questions falls after a document re-index, while replies get friendlier and better formatted. The blend cancels. The dashboard is flat and green.
Customers are being given wrong fee figures in a warm, well-structured tone. When it finally surfaces through complaints, the incident review asks when accuracy started declining — and the only available data is a blended score that never moved, so the start date cannot be established and the affected population cannot be bounded. The blend did not just hide the problem; it destroyed the evidence needed to remediate it.
3 · STAR for the interview
I needed the answer grader to tell me two different things: is this answer useful at all, and is every claim in it traceable to a source document. Those deserve different strictness. Refusing to show a partly useful answer is a worse outcome than showing it with a lower confidence label.
Get two independent verdicts without paying for two model calls, in a shape I could act on programmatically.
One call, two explicitly separated tasks in the prompt, with different rubrics. Task one is a deliberately lenient pass-or-fail on usefulness — the instruction says to fail only when there is genuinely nothing useful. Task two is a strict claim-level check that lists every claim the documents do not support, and classifies each one as arithmetic, an extrapolation from a visible trend, or general knowledge.
The output is a typed object with a grade restricted to two allowed values and a list of unsupported claims, each with a basis restricted to three allowed values. I map the count to a level deterministically: zero is grounded, up to three is partial, more than three is not grounded — which becomes high, medium or low confidence. The judgement is the model's; the threshold is mine and is in code, so it is auditable and changeable without touching a prompt.
One rubric detail with a privacy rationale: the judge must write a short label of at most eight words naming the metric and context — for example "BHP FY2024 EBITDA" — and must not reproduce the sentence or the value. The label is enough to re-retrieve against, and it keeps figures out of the field that gets logged and traced.
The two verdicts drive different behaviour. A fail on usefulness routes to the fallback path. Two or more unsupported claims trigger a targeted retry, using the claim labels as new retrieval queries and resetting the found-status of the companies named in those labels, so the retry looks specifically for the missing evidence rather than repeating the original search.
If I had asked for one blended score, none of that would be possible: I would have a number and no idea which of the two problems it represented.
- Constrain the shape with a schema, not a request
- Split judgements; blend only at the end
- Lenient on usefulness, strict on grounding
- Thresholds in code, judgement in the model
- Labels not values — keeps figures out of logs
- "How do you make an LLM judge's output usable in code?"
- "Why not just ask for a single quality score?"
- "How do you design a rubric?"
C · When you measure
Concepts 7–9A check nobody runs is not a check. This group is about placing each check where it will actually run, and about the fact that offline testing has a hard ceiling no amount of case-writing can lift.
Offline evaluation — and its unavoidable ceiling
"What can your offline suite never tell you?"
1 · The concept
The problem. You need to know whether a change is safe before customers see it. That means running against saved cases rather than live traffic.
What it is. Offline evaluation is running the fixed golden set and comparing output to recorded ground truth. Its virtues are exactly the ones the naive approach lacks: it is repeatable, so two runs are comparable; it is cheap enough to run often; it is safe, because no customer sees the output; and it is fast enough to fit inside a development loop.
The naive belief, and why it fails. The naive belief is that a large enough offline suite means the system is safe to ship. It fails for one reason that no amount of effort removes: an offline suite contains only failures somebody already imagined. Every case exists because a human thought of it. The failures that hurt you in production are, almost by definition, the ones nobody thought of — the question phrased in a way you did not anticipate, the customer who types three questions in one message, the product variant you forgot existed.
There is a second, subtler ceiling: offline cases are usually single turns. Real conversations have history, and the interesting failures often live in turn four, when the assistant has forgotten what "it" refers to or has carried a wrong assumption forward.
The real answer. Offline evaluation is a regression net, not a proof of safety. Its job is to guarantee that the specific things you already know can go wrong do not go wrong again. Every production incident becomes a new case, so the suite grows towards reality over time. And because it can never be complete, it must be paired with online measurement — concept 9 — which is the only instrument that can see failures nobody imagined.
A NAB-style example. Imagine a 200-case offline suite that passes 197. It contains fee questions, dispute flows, hardship phrasing and out-of-scope refusals. What it does not contain is a customer writing "my pay didn't go in and the rent comes out tomorrow, what do I do" — a message that is simultaneously a transaction question, a hardship signal and an urgent one. Nobody wrote that case because nobody thought of that phrasing. Offline says 197 out of 200. Offline is right and also not the whole picture.
Two ways to grow a suite past your own imagination — and the limits of each
- Mine real traffic. Cluster live questions, then sample from the clusters that have no coverage. This is the highest-value source because it finds the phrasings you would never invent. In a bank it needs privacy handling first: personal information has to be removed before the transcript can sit in a repository, and that constraint is a design input, not an afterthought.
- Adversarial and generated cases. Use a model to produce variations — rephrasings, typos, mixed-intent messages, prompt-injection attempts. Cheap and broad. The limitation is that a model generates the variations it finds natural, so it has its own blind spots, just different ones from yours. It widens coverage; it does not complete it.
Neither removes the ceiling. Both raise it, and the second one is cheap enough that there is no good reason to skip it.
2 · If this is missing
The team ships on a green offline suite with no online measurement behind it. A prompt change subtly breaks multi-turn behaviour: from turn three onward the assistant starts answering about the wrong account when a customer has more than one. Every offline case is a single turn, so the suite is still fully green.
For five weeks, customers with two accounts are told the balance of the wrong one. The first evidence is a complaint, not a metric. The incident review has to reconstruct which conversations were affected from raw traces, and the offline suite — which passed every single day of those five weeks — becomes evidence that testing was not adequate rather than evidence that it was.
3 · STAR for the interview
My whole evaluation setup is offline. 28 cases across four suites, plus 12 numeric retrieval cases, run from the command line.
Make offline runs cheap and specific enough that I would actually run them during development, rather than treating them as a ceremony.
The suites live in a registry — a dictionary mapping each suite name to its case-loader and its evaluator — so adding a suite is one entry, not a new script. The runner takes flags for which suite, a limit on how many cases, and one or more specific case ids, so I can run a single case while debugging it.
Every case prints PASS or FAIL with the specific
mismatch rather than just a verdict, then a per-suite Score: passed/total
and an overall total, and the process exits with a non-zero code if anything failed.
That last detail is what makes it usable by an automated system later.
Offline evaluation caught real regressions during development and gave me a vocabulary for talking about them — I could say "routing case 5 fails on the calculation flag" rather than "something feels off".
I am clear about the ceiling. Every one of those cases is one I wrote myself, so the suite tests my imagination. It is single-turn, so it cannot see conversation-state bugs. It has no live traffic behind it. In a production setting I would treat it as the regression net and put the weight of safety on online measurement, and I would add a rule that every incident becomes a permanent case.
- Offline = repeatable, cheap, safe, fast
- Ceiling: only failures someone imagined
- Mostly single-turn, so misses state bugs
- Registry dict; flags for suite, limit, id
- Non-zero exit code = usable by automation
- "What are the limits of offline evaluation?"
- "Would you ship on a green offline suite?"
- "How do you decide what goes into your test set?"
CI gating and cost tiering — three tiers, and the determinism rule
"How would you put this in a build pipeline?"
1 · The concept
The problem. Evaluation that runs when someone remembers to run it does not protect anything. It has to be automatic. But a full evaluation suite for a language system costs real money and takes real time, so running the whole thing on every commit is not affordable — and a slow pipeline gets bypassed, which is worse than no pipeline.
The naive approaches, and why both fail. Naive approach one: run the full real suite on every commit. It fails on cost and speed — engineers wait twenty minutes for feedback, and someone eventually adds a flag to skip it. Naive approach two: run nothing automatically and rely on the author to check. It fails because people are busy and the check is invisible; the suite rots quietly until nobody knows whether it still passes.
The real answer: tier by cost. Continuous integration should run a different amount of evaluation at each stage, matched to what that stage can afford to spend and how much confidence it needs to buy.
| Tier | Trigger | What runs | Cost / time | Blocks? |
|---|---|---|---|---|
| 1 · Mock | Every commit | Deterministic checks only. Case schemas validated, evaluator logic exercised, nothing external called. | Free, seconds | Yes — always |
| 2 · Smoke | Every pull request | A small real subset — a handful of cases per suite through the actual models — to prove the wiring, credentials and prompts still work. | Cents, ~2 minutes | Yes, on hard failures |
| 3 · Full | Nightly, scheduled | The complete real suite against a committed baseline, with the diff posted somewhere a human reads. | Dollars, ~20 minutes | Alerts, does not block |
The rule that makes this work: a release gate must be deterministic. A gate is a check that can stop a change from shipping. If the gate is non-deterministic — if it can fail on a re-run with no code change — then people learn to press retry, and after the third time they press retry without reading. At that point the gate has been trained out of existence while still appearing on the dashboard. So model-judged metrics can alert, and they can be reviewed by a human, but they must not be the thing that blocks a merge. What blocks a merge is the deterministic tier.
A NAB-style example. A change to the intent classifier's prompt. Tier 1 runs in eight seconds and confirms no case file is malformed and no evaluator crashes — free, on every push. Tier 2 runs twelve real cases on the pull request and blocks if the assistant now routes a hardship message to the transactions path, because that is an exact label comparison and a failure is unambiguous. Tier 3 runs overnight across 200 cases including groundedness judgements, and posts a diff against last night's committed baseline: three cases changed verdict, all in the same product area. A human reads that with their coffee. Nothing was blocked by a coin flip.
2 · If this is missing
With no automated evaluation in the pipeline, a refactor of the retrieval layer merges on a passing unit-test suite and a code review. The unit tests check that functions return the right shapes; none of them checks whether the right document comes back. The change is deployed the same day.
Retrieval recall drops from about 90% to about 60%. The assistant begins saying "I don't have that information" for questions it answered correctly last week. Because the deployment pipeline runs no evaluation, the first signal is a rise in contact-centre volume two days later. Diagnosis takes another day because nobody can say which deploy changed behaviour — there is no per-commit evaluation record to bisect against, so the investigation is a manual archaeology exercise rather than reading a table.
3 · STAR for the interview
Running the real suite costs API calls and takes minutes, so I did not want it firing on every save. But I also wanted a check I could run constantly without thinking about cost.
Build a free tier that still exercises real code, and a real tier I would only pay for deliberately.
I built two modes. Mock mode constructs a synthetic output shaped like a successful run — including fake retrieved documents carrying the expected company, fiscal year and metric values — and then runs it through the real evaluator code. It calls nothing external and costs nothing, and it genuinely catches problems: a malformed case file, a missing required field, a bug in an evaluator's comparison logic. It is deliberately not a proof that the retriever works, and I say so in the documentation.
Real mode takes the cheapest useful path rather than the most realistic one. Routing cases call the decision and rewrite nodes directly. Retrieval cases invoke only the retrieval subgraph. The full eleven-node graph is never run by the suite, because running it would multiply the cost without changing which stage the result pointed at.
The gap: none of this is in continuous integration. My only GitHub workflow deploys to an EC2 instance on push to main — fetch, hard reset, rebuild containers. There is no evaluation step anywhere in it, so a change that breaks routing deploys exactly as smoothly as one that fixes it.
The two modes gave me the right habit — mock constantly, real deliberately — and the mock tier is already exactly what a tier-1 gate needs to be: deterministic, free, seconds, and non-zero exit on failure.
Wiring it up is the smallest high-value change left in the project: one workflow step running mock mode on every push, which would block a broken case file today. A small real smoke tier on pull requests would follow, and the nightly full run needs the committed baseline from concept 12 before it means anything.
- Three tiers: mock every commit, smoke on PR, full nightly
- A gate must be deterministic or it gets retried away
- Judge metrics alert; they do not block
- Mock = synthetic output through real evaluators
- My gap: evals not in CI, only a deploy workflow
- "How would you integrate evaluation into CI/CD?"
- "How do you keep evaluation affordable at high commit frequency?"
- "Would you block a release on an LLM-judged score?"
Online evaluation — shadow, canary, A/B, and proxy metrics
"How do you evaluate with real users without risking them?"
1 · The concept
The problem. Offline evaluation cannot see failures nobody imagined, and it cannot tell you whether customers are actually helped. Only live traffic can. But live traffic is customers, and experimenting on customers of a bank has consequences.
The naive approach, and why it fails. The naive approach is to deploy the new version to everyone and watch the dashboards. It fails because the blast radius is 100% from the first second. By the time a problem is visible in aggregate metrics — which usually takes hours, because you need volume to see a change — every customer who used the assistant in that window has already had the degraded experience. In a hardship or dispute flow, some of those interactions are not recoverable by rolling back.
The real answer: three mechanisms, in increasing exposure.
- Shadow mode. The new version receives a copy of real traffic, its output is recorded, and nothing is shown to the customer. Exposure is zero. You get real questions and can compare old and new answers side by side offline. This is the cheapest way to discover the phrasings your golden set was missing. Its limit is that it cannot measure customer reaction, because nobody saw the output.
- Canary release. The new version serves a small slice of real traffic — start at 1%, then 5%, then 25% — with automatic rollback if a monitored metric breaks a pre-set bound. The name is from the caged bird taken into a mine: a small, early warning of a bad atmosphere. Exposure is bounded and known.
- A/B test. Two versions run side by side on randomly assigned traffic for long enough to compare a chosen metric with a stated confidence. This is the only one of the three that can establish that the new version is genuinely better rather than merely not visibly broken.
Proxy metrics, because live ground truth is absent. Offline you know the right answer. Live you do not — nobody labels the answers as they go past. So you measure behaviour that correlates with quality:
| Proxy metric | What it suggests | How it can mislead |
|---|---|---|
| Retry / rephrase rate | The customer did not get what they needed the first time. | Also rises when customers get curious and explore more deeply. |
| Refusal rate | The system is declining more than before. | A rise can be correct if traffic mix shifted towards out-of-scope questions. |
| Thumbs-down rate | Explicit dissatisfaction. | Very few people rate anything, and those who do are unrepresentative — usually the angry ones. |
| Abandonment | The customer left mid-conversation. | Some abandonment is success: they got the answer in one reply and left satisfied. |
| Escalation to a human | The assistant could not finish the job. Often the most trustworthy proxy, because it has a cost the customer chose to pay. | Escalation policy changes move it for reasons unrelated to answer quality. |
Stopping rules, agreed in advance. Before the experiment starts, write down what would make you stop it and what would make you accept it — the metric, the threshold, the duration, and who is allowed to call it. Then wire the stop condition to automatic rollback so it does not depend on someone being awake. The reason to pre-agree is not process theatre: once an experiment is running and you have invested in it, every number becomes arguable, and a threshold chosen after seeing the data is not a threshold.
A NAB-style example. A new answer-generation model for the assistant. Week one, shadow mode: real questions, no customer exposure, and comparing outputs reveals that the new model gives noticeably more directive language on mortgage questions — an advice-boundary risk that no offline case had caught. Fix, then week two, canary at 1%, with automatic rollback if escalation-to-human rises by more than three percentage points over a two-hour window, or if refusals on hardship-flagged conversations rise at all. Week three, a proper A/B on 50% of traffic with escalation rate as the primary metric and a pre-registered decision rule. Exposure grows only after each stage has produced evidence.
Why shadow mode is the highest-value first step — and what to watch for in a bank
Shadow mode is the only mechanism that gives you real user questions at zero customer risk. That combination is rare and it directly attacks the ceiling from concept 7: it hands you the phrasings you failed to imagine, which then become permanent golden-set cases.
Two cautions specific to a bank. First, running a shadow copy doubles the model spend for whatever share of traffic you mirror, so mirror a sample, not everything. Second, and more important, the shadow path touches real customer data. Its outputs, logs and traces are subject to the same privacy obligations as production — personal information must not leak into a comparison store that was set up quickly for an experiment. An experiment is not an exemption from the data handling rules.
There is one thing shadow mode cannot do: measure what customers do in response. For that you need exposure, which is what the canary is for.
2 · If this is missing
A model version upgrade goes to 100% of traffic on a Thursday afternoon on the strength of a green offline suite. The new model is more cautious in a specific way the suite never tested: when a message contains any hint of financial difficulty, it declines and suggests calling the contact centre.
Over four days, customers reaching out about hardship — people already in difficulty, at the moment they finally asked for help — are turned away by an automated system. Because there is no canary, exposure is total from minute one. Because there are no proxy metrics wired to alerts, the signal arrives as a spike in contact-centre wait times, not as a rollback. Because there is no pre-agreed stopping rule, the decision to revert takes a meeting. The rollback is technically five minutes of work and it happens on day four.
3 · STAR for the interview
My project is deployed — FastAPI backend, Next.js frontend, on an EC2 instance, with PostgreSQL checkpointing and LangSmith tracing configured — but it has no real user base.
Be straight about this: online evaluation is the entire category my project does not have.
My project has no online evaluation of any kind. No shadow mode, no canary, no production A/B, no drift detection, and no sampled human review of live answers. The A/B work I did is offline and component-level: two retrieval configurations over 12 saved cases, not two variants over live traffic. I want to be precise about that distinction, because "I ran an A/B test" would overstate it.
What I would build, in this order. First, sampled human review — take twenty live conversations a day and label them, because it needs no infrastructure and it is the fastest way to find out what I do not know. Second, shadow mode on a traffic sample, since it gives real questions at zero exposure and feeds the golden set. Third, a canary with automatic rollback wired to escalation rate and refusal rate. A full A/B framework last, because it is the most infrastructure for the least immediate safety.
One thing I do have that makes this cheaper to build: tracing is already wired up, so the per-run record needed to sample and label conversations exists rather than needing to be added.
No result to report, and I would say so directly rather than dress up the offline A/B as an online experiment. What I can offer is the design and the ordering, and the reasoning behind the ordering: cheapest thing that reveals unknown failures first, most infrastructure last.
- Shadow (0% exposure) → canary (1–25%) → A/B (50%)
- No live ground truth, so use proxy metrics
- Escalation rate is usually the most trustworthy proxy
- Stopping rules pre-agreed and wired to auto-rollback
- My gap: no online evaluation at all
- "How would you run an experiment on a live banking assistant?"
- "What do you measure when you have no ground truth in production?"
- "How do you limit the risk of shipping a model change?"
D · How you trust the result
Concepts 10–12The statistics group. The job ad asks for someone who can "quantify uncertainty" and "interpret results critically" — this is where that lives.
Variance and paired comparison
"How do you evaluate a system that gives different answers each time?"
1 · The concept
The problem. Run the suite twice with no code change and you get 82% and then 79%. Then you make a change and get 84%. Is 84 better than 82, or is it inside the range the system produces on its own?
Where the variation comes from. Naming the sources matters, because some are removable and some are not.
| Source | What it is | Removable? |
|---|---|---|
| Sampling | The model picks the next word probabilistically. Setting temperature to zero reduces this a lot. | Mostly, at a cost in answer quality and naturalness |
| Provider non-determinism | Even at temperature zero, hosted models are not bit-identical: batching, hardware and floating-point order all vary. | No |
| Model updates | The provider changes the model behind the same name. Your system changes without your code changing. | Only by pinning a version, where that is offered |
| Retrieval ties | Two documents with equal scores can come back in either order, and that order can change everything downstream. | Yes — sort by a stable tiebreaker |
| Judge variance | The grader is itself a model, so it can score the same answer differently. | Partly — structured output helps a lot |
| Concurrency | Parallel stages complete in different orders, so assembled context differs run to run. | Yes — order results deterministically before use |
The naive approach, and why it fails. The naive approach is one run per configuration, then compare the two totals. It fails because you are comparing two single samples from two noisy distributions and calling the difference an effect. With forty cases, three cases flipping is a 7.5-point swing — comfortably large enough to invent an improvement that does not exist, or to hide one that does.
The real answer, in two moves.
First: repeated runs are a measurement, not a nuisance. Run the same configuration three or five times and look at the spread. That spread is your noise floor, and it is the single most useful number in the whole exercise, because from then on any difference smaller than it is not evidence of anything. Most teams skip this and then argue about two-point differences for weeks.
Second: compare per case, not per mean. Do not compare 82% against 84%. For each individual case, record what the old configuration did and what the new one did, and count how many got better, how many got worse, and how many stayed the same. This is a paired comparison, and it is far more sensitive, because it removes the variation caused by some cases simply being harder than others. A net of plus-four with two regressions tells you something a mean never can: the change is a real improvement and it broke two things you should look at.
A NAB-style example. Two prompt versions for the assistant, both scoring 88% on 150 cases. The means say "no difference, ship either". Paired comparison says: 19 cases improved, 19 regressed. That is not "no difference" — it is a large behavioural change that happens to net to zero. And when you read which 19 regressed, they are all hardship-related. The means would have shipped that.
Comparing two means
82% → 84%. Two numbers, no idea how much of the gap is noise, no
idea which cases moved. Hides equal-and-opposite changes completely. Needs a large
sample to detect anything.
Paired per-case comparison
+5 better, −2 worse, 33 same. Names the direction, the size, and
exactly which cases to open. Detects the same effect with a far smaller sample, and
surfaces regressions the total conceals.
2 · If this is missing
A team compares two retrieval configurations, one run each, and sees 84% against 81%. They ship the 84% version. The true noise floor, had anyone measured it, is about four points — so the comparison was a coin flip, and in fact the shipped configuration is slightly worse.
Nothing dramatic happens. Retrieval quality degrades slightly and permanently, the team believes it improved, and the next comparison is made against this new and wrong baseline. Six months of that and the system is materially worse than it was, with a documented history of improvements. That is the most expensive failure on this page precisely because it never produces an incident — it just quietly wastes the whole team's judgement.
3 · STAR for the interview
I wanted to know whether a larger candidate pool plus reranking actually improved retrieval. My end-to-end answers varied run to run, so comparing two answers proved nothing.
Design the comparison so the variance was removed rather than averaged over.
Two design choices did the work. First, I removed the language model from the
measurement entirely — the A/B harness calls the retrieval node directly, so there
is no sampling variance in the loop at all and the only difference between the two
runs is retrieval configuration. Second, I made the comparison strictly paired:
every one of the 12 numeric cases runs under both configurations, and the harness
prints a per-case verdict — same, NEW FOUND IT,
NEW LOST IT, or rank 6 -> 2 better.
I report the aggregate too — hit@top_k, hit@5 and
mean_rank — but the per-case column is what I actually read, because
NEW LOST IT is a regression that a stable mean would have hidden
completely.
A comparison where a rank change of one position is a real signal rather than noise, because the noise was designed out instead of averaged out. That is the general principle I would carry into a bank: before adding repeat runs, ask what variance you can eliminate from the measurement.
The honest gap: I run each case once. With no language model in the retrieval A/B that is defensible, since the path is essentially deterministic. For anything end-to-end it is not defensible, and the first thing I would add is three repeats of the same configuration to establish a noise floor — before comparing anything to anything.
- Name the sources; eliminate what you can
- Repeat runs = your noise floor, a measurement
- Ignore any difference smaller than the floor
- Pair per case: better / worse / same
- Equal means can hide a huge behaviour change
- "How do you evaluate a system that is not deterministic?"
- "How do you know a 2% improvement is real?"
- "Where does non-determinism in your pipeline come from?"
Sample size and uncertainty — direction, not significance
"You have forty cases. What can you actually conclude?"
1 · The concept
The problem. Every evaluation result is a number computed from a sample. A different sample would give a different number. How wrong could it be, and what claims does this many cases actually support?
The naive approach, and why it fails. The naive approach is to report the number as though it were the truth: "our system is 84% accurate". It fails because on forty cases, 84% has an uncertainty band of roughly plus or minus twelve percentage points. The honest statement is "somewhere between about 72% and 93%", which is a very different sentence — and it makes clear that comparing 84% against 81% on forty cases is comparing two heavily overlapping ranges.
A rough guide worth memorising. For a proportion around 50%, the uncertainty band is roughly plus or minus one over the square root of n. That gives a useful mental table:
| n | Rough band on a proportion | What it supports |
|---|---|---|
| 12 | ≈ ±29 points | Nothing quantitative. Useful only as a paired per-case comparison, where you read which cases moved. |
| 40 | ≈ ±16 points | Direction, if the shift is large and paired. Not a rate you would put in a report. |
| 100 | ≈ ±10 points | Direction reliably; large effects with confidence. |
| 400 | ≈ ±5 points | A rate you could reasonably quote. |
| 1000+ | ≈ ±3 points | Small effects; the scale online experiments operate at. |
The real answer: say "direction, not significance". With a small suite, do not claim a rate and do not claim statistical significance. Claim a direction, from a paired comparison, and say what n it rests on. "On 12 paired cases, four improved, none regressed" is a legitimate and useful statement. "Retrieval accuracy improved by 33%" from the same data is not. Knowing which sentence your data supports is the whole skill.
Pre-register the reject criterion. Before running the comparison, write down what result would make you reject the change — for example: "I reject if any case regresses from found to not-found, or if the net is below plus two." The reason is human, not mathematical. After you see the data, you will find a story for whatever it shows, especially if you built the thing being tested. Writing the rule first is what separates an experiment from a demonstration.
A NAB-style example. Two versions of the assistant tested on 50 hardship cases. Version B refuses correctly 46 times, version A 42 times. That looks like a clear win, and reported as "8 points better on refusal correctness" it would ship. With 50 cases the band on each is roughly ±14 points, so the two ranges overlap almost entirely — the paired view might show 6 improvements and 2 regressions, a net of plus 4 on 50 cases, which is a direction worth pursuing and not a measured improvement. The correct next step is more cases in that category, not a decision.
Statistical significance, and why it is often the wrong question here — plus the trap of testing many metrics
Statistical significance answers "could this difference plausibly be chance?" It does not answer "is this difference big enough to care about?" A significant improvement of 0.3 points is significant and irrelevant. On a forty-case suite, only an enormous effect would ever reach significance, so the concept mostly does not apply — which is why the honest framing at that scale is direction plus paired detail.
The trap to know about: if you check twenty metrics, roughly one will look significant by chance alone at the conventional threshold. That is how teams discover exciting improvements that evaporate on the next run. The defences are to name one primary metric before you start, treat everything else as exploratory, and be openly suspicious of a win found in the fifteenth metric you looked at.
Related, and worth saying out loud in an interview: a pre-registered primary metric is also what stops an experiment being stopped early the moment it happens to look good. Peeking repeatedly and stopping on a favourable reading inflates the false-positive rate substantially, even when every individual calculation is done correctly.
2 · If this is missing
A team reports to a governance forum that the assistant "answers correctly 91% of the time", measured on 45 cases. The true band is roughly plus or minus fourteen points. The 91% figure enters a slide, then a risk register, then a control description, and eventually becomes the number the bank believes about the system.
Later, real performance turns out to be nearer 78% — entirely consistent with the original measurement, which was never wrong, only over-claimed. The problem is now a governance one: a control was documented on the basis of a number that the evidence never supported, and the bank has to explain to an auditor how that figure was derived and why nobody stated its uncertainty. The engineering was fine. The sentence was not.
3 · STAR for the interview
My suite is small: 28 cases across four suites, 12 numeric retrieval cases, one run per case. That is a real constraint on what I am entitled to claim.
Match my claims to my sample size, and design the comparison so a small sample was still informative.
I never reported a rate from these suites as a quality figure. What I report is
the paired per-case verdict, which is the strongest thing 12 cases can support. The
design choice that made 12 enough was pairing plus removing the language model:
with the same cases under both configurations and no sampling variance, a
NEW LOST IT is unambiguous evidence about that case, even though 12
cases cannot support a claim about a rate.
The reject criterion I used was effectively "any case going from found to
not-found is a problem I must explain, regardless of the aggregate". I would now
write that down before running rather than holding it in my head, and I would name
one primary metric in advance — hit@5 — instead of reading three and
picking the encouraging one.
Conclusions I can still defend, because they were stated as direction on a named n rather than as a measured rate. If asked "is your system 90% accurate", the correct answer is that I do not know, and that 40 cases could not tell me.
What I would fix: n is the binding constraint on everything. I would grow the suite from real traffic towards a few hundred cases per category, prioritising the categories where a mistake is expensive — for a bank, refusals and anything touching advice — rather than growing it uniformly.
- Band ≈ ±1/√n — 40 cases means ±16 points
- Say direction, not significance
- Pairing buys sensitivity a small n cannot
- Pre-register the primary metric and reject rule
- 20 metrics → one false win by chance
- "How many test cases do you need?"
- "How do you quantify uncertainty in an evaluation result?"
- "Your suite is small — what does that let you conclude?"
Drift detection and regression baselines
"Your code has not changed in a month. Can behaviour still change?"
1 · The concept
The problem. Ordinary software only changes when you change it. This kind of system does not. Behaviour can change with no code change at all — drift — for several independent reasons.
- Model drift. The provider updates the model behind the same name. Your prompts now meet a slightly different model.
- Data drift. The questions change. A new product launches, a rate rises, a news story lands, and suddenly a third of traffic is about something your system was never tested on.
- Index drift. Documents are re-ingested, a chunking parameter changes, and what the retriever can find changes underneath a prompt that did not move.
- Dependency drift. A library upgrade changes tokenisation, scoring or parsing behaviour in a way nobody read the changelog for.
The naive approach, and why it fails. The naive approach is to watch the average of your quality metric on a dashboard. It fails because averages are remarkably good at hiding the failures that matter. If a system is excellent for 90% of questions and catastrophic for a new 10%, the average moves by a few points and looks like ordinary fluctuation. The information you needed was in the shape of the distribution, not its centre.
The real answer, in two parts.
Monitor distributions, not just averages. Watch the shape: the share of answers labelled low confidence, the tail of slow responses rather than the median, the mix of question categories arriving, the refusal rate broken down by category. A change in composition — the same average made of different parts — is the early warning that an average by construction cannot give you.
Commit the baseline to the repository. Store last known-good per-case results as a file under version control, and have the nightly run diff against it rather than print a number. Then a regression is a reviewable change, with a date, an author and a reason — and re-baselining becomes a deliberate act someone approves, instead of a number quietly moving in a log nobody reads.
A NAB-style example. Nothing is deployed for six weeks. The model provider updates the underlying model. The assistant becomes slightly more willing to answer product-comparison questions with a recommendation — drifting towards personal advice. Average quality is unchanged, possibly improved, because the answers are more helpful-sounding. With a committed baseline, the nightly run shows eleven cases changing verdict on the advice-boundary check on one specific date, and the date points straight at the model update. With only an average on a dashboard, this is invisible until a compliance review months later.
What to put in a committed baseline — and how to make re-baselining honest
Per case: the case id, the verdict, the primary metric value, and a short reason on failure. Plus a header recording when it was produced, which model versions were used, and which commit produced it. That header is what lets you attribute a change six weeks later.
What not to store: the full answer text. It changes every run because the model is non-deterministic, so the diff becomes unreadable and people stop reading it. Store the verdict and the metric — the things that are supposed to be stable.
Re-baselining needs a rule or it becomes a way of erasing regressions. A workable one: the baseline file may only be updated in a commit that does nothing else, with a message stating which cases changed and why, and approved by someone who did not write the change. In a bank this doubles as the audit artefact — it is a dated, authored, reviewable record of when behaviour changed and who accepted it, which is exactly what a regulator asking for reconstructable decisions wants to see.
2 · If this is missing
Scores print to a console and nowhere else. In March someone records "38/40" in a chat message. In June the suite scores 34/40. Nobody can tell whether the four failures are new, or which four they are, or whether they were already failing in March, because the March run left no per-case record.
The investigation becomes archaeology: re-running old commits, guessing at model versions that are no longer available, and eventually concluding that behaviour probably changed sometime in April. If those four cases are refusal or advice-boundary cases, the bank cannot establish when the behaviour started, so it cannot bound the affected customer population and must treat the whole window as potentially affected. That is the difference between a one-page note and a remediation programme, and it is caused entirely by storing a number in a log instead of a file.
3 · STAR for the interview
I hit drift for real, from the least glamorous source: a library default. My keyword search used a standard BM25 implementation, and its behaviour on my corpus was not what I assumed.
Work out why obviously relevant documents were not coming back, when the code was doing exactly what I had written.
I measured instead of guessing. The library computes a term weight that goes negative for very common words, and to avoid negative scores it clamps every negative value up to a floor — a quarter of the average, which on my corpus was 1.811. The word "the" appeared in 84% of my chunks, 5,901 of 7,061, so its true weight was −1.626 and it was clamped to 1.811 — higher than "Group" at 1.542, which is a genuinely discriminating word in mining reports.
Only 8 words were affected, which is why it looked harmless. But those 8 words appear in nearly every natural-language question. For "what was the revenue of the group in 2024", 59% of the top document's score came from what, was, the, of, in. After removing stop words, the new top five documents had zero overlap with the old top five. Normalisation moved the documents matching "revenue" from 221 to 278, up 26%, and cut the vocabulary from 37,933 to 23,676, down 37.6% — where the value was that document frequencies became correct, not that memory was saved.
The fix introduced a bug, which is the part I would tell an interviewer. A query made entirely of stop words — "what is it" — tokenised to nothing. An empty query scores every document 0.0, and the library returns an arbitrary 20 documents with no error at all. Silent, plausible-looking, wrong. I fixed it inside the tokeniser by falling back to the unfiltered tokens, with an assertion in a self-check so it cannot regress unnoticed.
A retrieval layer whose scoring I could explain, and a permanent lesson: a library default is a behavioural dependency, and drift can arrive through an upgrade nobody thought was risky.
The gap I would fix first in this whole concept: my project has no committed baseline. Scores only print to the console. Nothing is stored, so nothing can be diffed, and I could not today prove that behaviour has not drifted since I last ran the suite. That is a small file and a nightly job, and it is the highest value-per-hour item left on my list.
- Behaviour changes with no code change
- Four sources: model, data, index, dependency
- Monitor distributions and mix, not averages
- Baseline in a file, not a log; diff nightly
- Re-baselining = its own reviewed commit
- "How do you detect that a model has changed underneath you?"
- "How do you prevent silent regressions over time?"
- "Tell me about a time you debugged something surprising."
What my project does not have
Say this before they find itVolunteering this list is stronger than being caught by it. The framing to use: "here is what it has, here is what it does not, and here is the order I would fix it in."
| Missing | Concept | Why it matters | Priority to fix |
|---|---|---|---|
| Judge validation | 5 | Two model graders run at runtime and neither has ever been measured against human labels. No agreement rate exists, so every groundedness verdict is uncalibrated. Both fail open, so a broken grader reads as "all fine". | 1 — one day of labelling |
| Committed baseline | 12 | Scores only print to the console. Nothing is stored, so nothing can be diffed and no drift can be attributed to a date. | 2 — a file and a nightly job |
| Evals in CI | 8 | The only GitHub workflow deploys to EC2 on push to main. A change that breaks routing ships as smoothly as one that fixes it. Mock mode is already free, deterministic and exits non-zero — it is one workflow step away from being a gate. | 3 — one step |
| Online evaluation | 9 | None of it: no shadow mode, no canary, no production A/B, no drift detection, no sampled human review of live answers. So the failures nobody imagined are entirely invisible. | 4 — needs users first |
| Latency and cost per query | 3 | Not recorded anywhere, so no quality-versus-cost trade-off can be argued with evidence. | 5 — nearly free to collect |
| Page-filter recall | 3 | The vision-ingest page filter is a heuristic based on numeric density and its recall has never been measured. It is the one pipeline stage with no measurement, and it can silently drop the page containing the answer. | 6 — needs labelled pages |
| Adequate sample size | 11 | 28 suite cases plus 12 numeric, one run per case, all hand-written by me. Big enough for direction on paired comparisons, not for any claimed rate. | Ongoing |
Cheat sheet · twelve rows
Read this lastOne row per concept. The last column is a sentence you can say verbatim. Four groups of three: what you measure, how you measure, when you measure, how you trust it.
| Concept | One-line definition | If missing | The sentence to say |
|---|---|---|---|
| 1 · Ground truth and the golden set | The correct answer written down in advance, stored in version control as fixed input-and-answer pairs. | Scores are opinions that drift; you can never prove you have not gone backwards. | "I wrote the expectation down before running anything. 28 cases in four suites, five required fields, and the loader raises if any is missing." |
| 2 · Component vs end-to-end | Grade the final output for user-visible quality; grade one stage in isolation to find the fault. | You know something broke and never which stage — and stage errors cancel out invisibly. | "End-to-end is my release signal, component-level is my debugging instrument. My retrieval A/B calls the node directly with no LLM in the loop." |
| 3 · Per-stage metrics | Routing accuracy, retrieval recall@k, groundedness rate, refusal correctness in both directions, latency and cost. | A blended score can rise while over-refusal quietly harms customers in hardship. | "One metric per stage, matched to that stage's job — and refusal errors as two numbers, because a wrong refusal and a wrong answer have completely different costs." |
| 4 · Deterministic vs LLM judge | Three tiers: exact comparison, pattern and numeric checks, then a model judge only where meaning must be understood. | Evaluation becomes slow, costly and non-repeatable, so real regressions get dismissed as noise. | "Never send to a model what a regex or a numeric comparison can settle. My production guardrail is a ten-character length check with no LLM at all." |
| 5 · Judge validation | Measure agreement between the judge and human labels; expect length, self-style and position bias. | You optimise a bias in your instrument for weeks and the score rises the whole time. | "The judge is my measuring instrument, so it needs calibrating. I have never validated mine against human labels — that is the biggest gap in my project and the first thing I would fix." |
| 6 · Rubrics and structured verdicts | Constrain the judge's output to a typed schema; ask separate questions instead of one blended score. | Opposite movements cancel inside one number, destroying the evidence needed to remediate. | "One call, two separated tasks: a lenient pass on usefulness and a strict claim-level groundedness check. Thresholds live in code, judgement in the model." |
| 7 · Offline evaluation | Fixed saved cases, repeatable and safe — but containing only failures someone already imagined. | A fully green suite while a multi-turn bug affects customers for five weeks. | "Offline is a regression net, not a proof of safety. Every case in mine is one I imagined, and it is single-turn, so it cannot see conversation-state bugs." |
| 8 · CI gating and cost tiering | Mock every commit, small real smoke on pull requests, full real suite nightly against a committed baseline. | A refactor drops recall from 90% to 60% and the first signal is contact-centre volume two days later. | "A release gate must be deterministic, or people retry it until it passes. Judge metrics alert, they do not block. My evals are not in CI yet — mock mode is one workflow step away." |
| 9 · Online evaluation | Shadow at zero exposure, canary at 1–25% with auto-rollback, then A/B; proxy metrics because live ground truth is absent. | Hardship customers turned away for four days at 100% exposure before anyone notices. | "Shadow first because it gives real questions at zero risk. Proxy metrics — retries, refusals, thumbs-down, abandonment, escalation — with stopping rules agreed before the experiment starts. My project has none of this." |
| 10 · Variance and paired comparison | Repeat runs to establish a noise floor; compare case by case rather than comparing two means. | You ship a slightly worse system believing you improved it, and re-baseline onto the mistake. | "Before averaging noise away, I ask what variance I can eliminate. I took the LLM out of the retrieval A/B, then paired every case, so a one-rank move is real signal." |
| 11 · Sample size and uncertainty | Band is roughly ±1/√n; claim direction, not significance; pre-register the reject criterion. | A 91% figure from 45 cases becomes a documented control the evidence never supported. | "On 40 cases the band is about ±16 points, so I report direction from a paired comparison and name the n. I would not quote a rate from that." |
| 12 · Drift and regression baselines | Behaviour changes with no code change; monitor distributions; keep a committed baseline file, not a number in a log. | Four failures appear between March and June and nobody can say which, or when, or bound who was affected. | "Drift arrives through the model, the data, the index or a dependency. Mine arrived through a BM25 library default that clamped stop-word weights above real terms — 59% of the top document's score came from five stop words." |
Glossary
54 termsEvery term and abbreviation used on this page, in the order a reader is likely to need it. 54 entries, split into two folded sets.
Core terms · 40 entries — measurement, methods, and timing
- LLM (large language model)
- A model trained on text that generates text. The component that writes the answers, and in this page also sometimes the component that grades them.
- Probabilistic system
- A system whose output is sampled rather than computed, so the same input can produce different outputs. The reason ordinary testing does not transfer.
- Non-determinism
- The property that repeating an operation does not repeat the result.
- Temperature
- A setting controlling how randomly a model picks its next word. Zero means always take the most likely word, which reduces but does not remove variation.
- Ground truth
- The recorded correct answer for a test input, decided in advance by a human or taken from an authoritative source.
- Golden set
- A fixed, version-controlled collection of test inputs each paired with its ground truth. Also called a gold set or golden dataset.
- Case
- One test item: an input plus its expected properties. In my project, one line of a JSON lines file.
- JSONL (JSON lines)
- A file format with one complete JSON object per line. Easy to append to and to review in a version-control diff.
- Suite
- A named group of cases that share an evaluator. Mine are routing, retrieval, calculation and boundary.
- Evaluator
- The code that compares an output against a case's expectations and returns the list of failures.
- End-to-end evaluation
- Sending an input through the whole system and grading only the final output. Measures the user experience; cannot localise a fault.
- Component-level evaluation
- Testing one stage in isolation with a controlled input. Localises faults and is far cheaper.
- Pipeline / stage
- A system built as a sequence of steps, each transforming the work in progress. A stage is one such step.
- Supervisor graph
- A control structure where one node decides which node runs next, rather than a fixed straight line. Mine has eleven nodes with a five-node retrieval subgraph inside.
- Node
- One unit of work in such a graph — for example the routing decision, the query rewrite, or the answer grader.
- Routing accuracy
- The share of inputs sent down the correct path.
- Recall
- Of everything that should have been found, the fraction actually found.
- Precision
- Of everything returned, the fraction that was actually correct.
- Recall@k
- Recall measured only within the first k results, where k is however many get passed to the next stage.
- Groundedness
- Whether every factual claim in an answer can be traced to a retrieved source document.
- Refusal correctness
- Whether the system declined the right things, reported as two separate numbers: wrong refusals, and wrong answers to things it should have refused.
- Latency
- Wall-clock time from question to reply.
- Regular expression (regex)
- A compact pattern language for matching text. The cheapest tier-2 evaluation instrument.
- LLM-as-judge
- Using a language model to grade another model's output, given the input, the output, and a rubric.
- Pointwise / pairwise judging
- Pointwise scores one answer against a rubric. Pairwise compares two answers and picks the better, which is more reliable but suffers from position bias.
- Rubric
- A written definition of what each verdict means, with the specific conditions for each, so two graders reach the same answer.
- Structured output
- Forcing a model to return a typed object with named fields and allowed values, enforced by a schema rather than requested politely.
- Agreement
- How often two labellers give the same verdict on the same item. Raw agreement is the plain percentage.
- Cohen's kappa
- Agreement adjusted for how often two labellers would agree purely by chance. Necessary when one verdict is much more common than the other.
- Length bias
- The tendency of a model judge to prefer longer answers regardless of quality.
- Position bias
- The tendency of a model judge to favour one slot in a pairwise comparison. Fixed by running both orders and counting only when they agree.
- Fail open / fail closed
- Fail open means that when a check errors, the work is allowed through. Fail closed means it is blocked. Both of my model graders fail open.
- Offline evaluation
- Running fixed saved cases away from live traffic and comparing against recorded ground truth.
- Online evaluation
- Measuring on live traffic, where no ground truth exists and behaviour must be inferred from what customers do.
- CI / CD (continuous integration / delivery)
- An automated system that runs checks on every code change, and the part that then ships it. A gate is a check that must pass before a change proceeds.
- Shadow mode
- The new version receives a copy of real traffic and its output is recorded but never shown to a customer. Zero exposure.
- Canary release
- Serving a small, bounded slice of real traffic with automatic rollback if a monitored metric breaks a pre-set limit.
- A/B test
- Two versions run on randomly assigned traffic long enough to compare a chosen metric with a stated confidence.
- Proxy metric
- An observable behaviour used as a stand-in for quality when ground truth is absent: retries, refusals, thumbs-down, abandonment, escalation to a human.
- Paired comparison
- Evaluating both variants on the very same items and comparing item by item rather than comparing two averages. Far more sensitive at small sample sizes.
Second set · 14 entries — statistics, drift and project-specific terms
- Noise floor
- The spread you get from repeating the same configuration with no change. Any difference smaller than it is not evidence.
- Statistical significance
- How surprising a result this large would be if there were truly no difference. Says nothing about whether the difference is big enough to matter.
- Pre-registration
- Writing down the primary metric and the accept-or-reject rule before running the experiment, so the threshold is not chosen after seeing the data.
- Multiple comparisons
- The problem that checking many metrics makes a chance-looking win nearly certain. Roughly one in twenty at the conventional threshold.
- Drift
- Behaviour or incoming data changing over time with no code change. Comes from the model, the data, the index, or a dependency.
- Regression baseline
- A committed file of last known-good per-case results, diffed by each run so a regression becomes a reviewable change with a date and an author.
- BM25
- A long-standing keyword search scoring method: a document scores higher when it contains the query's words often and those words are rare across the corpus.
- IDF (inverse document frequency)
- The rarity part of that score. A word in almost every document carries almost no information, and its raw value can go negative.
- Stop words
- Extremely common words — the, of, is, what — that carry little meaning for search. The eight clamped words in my corpus were of this kind.
- Cross-encoder reranker
- A second-pass model that reads the query and each candidate document together and reorders them. More accurate than first-pass search and too slow to run over a whole corpus.
- hit@k / mean_rank
- In my A/B harness: how many cases had a correct document within the first k results, and the average position of the first correct document.
- Mock mode
- Building a synthetic successful-looking output and running it through the real evaluator code. Validates case schemas and evaluator logic, calls nothing external, costs nothing.
- Personal financial advice
- In Australia, a recommendation about a specific person's circumstances. Licensed activity, so an assistant must not give it. General information is permitted.
- Hardship support
- The process for customers who cannot meet repayments. A wrong refusal here is a real harm to a vulnerable customer, not merely a poor answer.