Working out what the user actually wants

What it actually is

Also called intent classification. The plainest possible definition: deciding what the user really wants the system to do.

Start with the problem it solves. A person types a sentence. A sentence is just text — a sequence of characters. Software cannot execute a sentence. Software executes functions: get_order_status(order_id), cancel_order(order_id). So something has to stand between the sentence and the function and decide which function this sentence is asking for. That something is intent recognition.

The important part is that it is not keyword matching. Look at three different sentences:

"Can you check where order 123 has got to?"
"Why hasn't my parcel arrived yet?"
"What's the status on this order?"

They share almost no words. One of them doesn't contain the word "order" in the way the others do; one is phrased as a complaint rather than a request. But all three want the same thing to happen, and that thing has a name:

check_order_status

That name is the intent. Once the agent has it, it knows what comes next:

check_order_status
   ↓
Order API → get_order_status(order_id)

Note the phrase "executable task category". An intent is not a topic. "Orders" is a topic. cancel_order is a task category, because there is a specific thing the system does when it sees that label. That distinction turns out to drive the entire design, and we come back to it in Start from business actions.

Where it sits in an agent

Intent is one step in a chain. Knowing the chain is how you answer "walk me through your agent" without waffling.

Here is the shape of a typical agent turn. Read it top to bottom — each step depends on the one above it:

User input
   ↓
Understand the request
   ↓
Intent recognition          ← what does the user want?
   ↓
Extract parameters / entities   ← what values does that need?
   ↓
Anything missing?               ← if yes, ask
   ↓
Choose tool / workflow
   ↓
Execute
   ↓
Generate the answer

Worked through one real sentence:

Two different things: intent and slots

These get confused constantly, so pin them down separately.

Both together, from one sentence — "Cancel the iPhone I ordered yesterday":

Intent
  cancel_order

Slots
  date    = yesterday
  product = iPhone

Notice that the slots here are not yet usable. "Yesterday" is not a date the API accepts, and "iPhone" is not an order ID. Resolving those — turning loose human values into system values — is its own step, and a common source of failures that get blamed on intent recognition when they are really slot-filling failures.

Why an agent needs it at all

Because an agent has many capabilities, and only one of them is correct.

Imagine a customer-service agent holding these tools:

get_order()
cancel_order()
create_return()
check_inventory()
update_address()
create_support_ticket()

The user writes five words: "My thing hasn't arrived yet." Before anything can happen, the agent has to commit to an interpretation:

Is this...
  A. checking the order?
  B. checking the delivery?
  C. asking for a refund?
  D. making a complaint?

There is no neutral option. Whatever it picks determines which tool runs. And here is why this matters more than it first appears:

Concretely:

So a large part of any agent architecture is really this chain:

Intent
   ↓
Policy / workflow
   ↓
Tool

Intent picks the policy; the policy picks the tool. Keeping those three separate is what lets you change a business rule without retraining anything.

Start from business actions, not from language

The most common mistake is designing dozens of intents on day one. The second most common is naming them after topics.

The instinct is to look at user messages and group them by what they sound like. That produces topic labels. Do the opposite: write down what the system is able to do, and let the intents fall out of that. For an e-commerce assistant, the capability map might be:

Orders
 ├─ look up an order
 ├─ modify an order
 └─ cancel an order

Delivery
 ├─ check delivery status
 └─ change delivery address

Refunds
 ├─ request a refund
 └─ check refund status

Account
 ├─ update details
 └─ reset password

Now the intents are just names for those leaves:

order_status
cancel_order
modify_order
delivery_status
change_delivery_address
request_refund
refund_status
reset_password

Why topic names fail

Compare two candidate label sets:

The label order is useless to the agent because it does not answer the only question that matters — what do I do now? Four completely different things live inside it: checking, cancelling, modifying, reordering. One of them is read-only and harmless; another is irreversible. Collapsing them into one label throws away exactly the information the next step needs.

Choosing the granularity

The single most useful test in intent design, and it takes one question to apply.

Suppose users ask both "check my order" and "check my delivery". Do you need one intent or two?

check_order

        ...or...

check_order_status
check_delivery_status

The tempting answer is to decide based on how different the sentences sound. That is the wrong axis entirely. The right question is:

Applied:

The second case is worth dwelling on, because it explains why more intents is not safer. Every additional label is another opportunity for misclassification, another set of examples to write, another row and column in the confusion matrix. A distinction that changes nothing downstream is pure cost.

What a proper intent definition contains

A name on its own is not a definition. The boundaries are the definition.

Writing refund in a list and moving on is how you end up with a classifier that behaves unpredictably — because nobody, human or model, knows precisely what belongs in it. A complete definition looks like this:

Intent name
  request_refund

Description
  The user is explicitly asking for money back on an
  order they have already paid for.

Include
  "I'd like a refund"
  "Can I return this order?"
  "Cancel it and refund me"

Exclude
  "When will my refund arrive?"
  "Why haven't I received my refund?"

The Include and Exclude sections do the real work. Definitions fail at their edges, not in their middle — nobody misclassifies "I want a refund". The failures live in the neighbouring pair:

"Why hasn't my refund arrived yet?" contains the word refund and sounds unhappy, so a careless system files it under request_refund and tries to issue a second refund. It is refund_status. One is a query; the other is a transaction. Writing that boundary down explicitly, with examples on both sides, is the cheapest accuracy improvement available in intent design — and it doubles as your test set.

Three ways to actually build it

Classical classifier, LLM classifier, or no explicit classifier at all. All three are current — the third is where most new systems land.

User text
   ↓
classifier
   ↓
refund_status:   0.85
request_refund:  0.10
other:           0.05

You are an intent classifier.

Available intents:
  1. check_order_status
  2. cancel_order
  3. request_refund
  4. refund_status
  5. change_address
  6. unknown

Return JSON only.

{
  "intent": "refund_status",
  "confidence": 0.94
}

Tools:
  get_order_status
  cancel_order
  request_refund
  get_refund_status
  change_address

User
 ↓
LLM
 ↓
Tool selection
 ↓
Tool execution

Hierarchical routing: don't classify forty things at once

For anything enterprise-sized, two coarse decisions beat one fine-grained one.

Asking a classifier to choose between forty labels in a single shot is fragile: the labels start overlapping, the examples blur, and accuracy degrades in ways that are hard to attribute. Split the decision into layers instead. The top layer answers only "what kind of thing is this":

                 ┌─ knowledge_query  → retrieval / RAG
                 │
User → Router ───┼─ business_action   → workflow / tool
                 │
                 ├─ conversation      → plain LLM reply
                 │
                 └─ unknown           → ask for clarification

Only inside business_action do you make the fine-grained choice:

business_action
       ↓
order      refund       delivery
 ├ check    ├ request     ├ status
 ├ cancel   └ status      └ change_address
 └ modify

This is called hierarchical intent classification. Three reasons it holds up better:

• Each decision has fewer options. Four-way then five-way is far more reliable than one twenty-way choice.
• The dangerous boundary is guarded early. "Is this a question or an action?" is the split that decides whether anything can change state — and it's the easiest one to get right.
• It localises failures. When accuracy drops you can see which layer moved, instead of staring at one number.

The two intents everyone forgets

Unknown, and ambiguous. Leaving them out is what turns a classifier into a liability.

Unknown / out of scope

A classifier asked to choose between five labels will always choose one of the five. That is what classification means. So if your agent knows only cancel_order, check_order and refund, and a user asks "who is your CEO?", the model does not have the option of saying "none of these" — unless you gave it one.

unknown
out_of_scope
ambiguous

Without those, an off-topic question gets forced into a business intent, and the agent calls a tool for a request that had nothing to do with it. This is the mechanism behind a whole class of embarrassing production incidents, and it is prevented by adding one label and one instruction.

Ambiguous — the case for not guessing

The user says: "I need to change my order." Change what? Any of these is plausible:

change_address
change_quantity
change_product
cancel_order

The wrong response is to pick the most likely one and act. The right response is to notice that no option is clearly ahead and say so:

intent     = modify_order
confidence = low
   ↓
"Sure — did you want to change the delivery
 address, the quantity, or something else
 about the order?"

That asymmetry is the whole idea: the cost of asking one extra question is small and recoverable, and the cost of cancelling the wrong order is neither.

When one message contains two intents

Real users do not send one instruction at a time. This is where intent recognition turns into planning.

"Check my order, and if it hasn't shipped yet, cancel it."

There are two actions in that sentence, and the second is conditional on the result of the first. A system built on the assumption of one message = one intent has no way to represent it; it picks whichever it thinks is dominant and drops the other half silently — which is worse than failing loudly.

{
  "intents": [
    { "intent": "check_order_status" },
    { "intent": "cancel_order",
      "condition": "if not shipped" }
  ]
}

From which the agent builds a small plan:

get_order_status
       ↓
if status != shipped
       ↓
cancel_order

You do not have to support this from day one. But you should know where the ceiling of your design is, and be able to say what the system currently does with such a message — which is a question interviewers like, because the honest answer reveals whether you have actually run one of these in production.

Evaluating it, starting with accuracy

Accuracy is where you begin and it is nowhere near where you stop.

The starting point is a test set: inputs paired with the label you believe is correct, called the ground truth.

Run the model and compare:

Accuracy is then the simplest possible summary — how many did it get right, over how many there were:

Accuracy = correct / total

950 / 1000  =  95%

And that number, on its own, is close to meaningless for an agent. The next three sections are why.

Precision, recall, F1

Accuracy averages over everything. These look at one intent at a time — which is where the damage actually is.

The problem with a single accuracy figure is that it hides distribution. If 90% of your traffic is order_status and the model nails it, you can score 95% overall while being hopeless at request_refund — the one that moves money. So measure per intent.

Precision = 90 / 100 = 90%

Recall = 90 / 120 = 75%

      2 × Precision × Recall
F1 = ────────────────────────
        Precision + Recall

The confusion matrix tells you what to fix

Aggregate metrics tell you something is wrong. This tells you which pair of intents is bleeding into which.

A confusion matrix is a grid: rows are what the input truly was, columns are what the model said. The diagonal is correct; everything off it is a specific, named mistake.

Read the middle row. Of 100 genuine refund_status questions — people asking where their money is — the model called 20 of them request_refund. That is not a vague quality problem. It is one specific leak, in one direction, and it is the dangerous direction: a status question being treated as a transaction request.

It also tells you the fix, which is not "train more" but "define better":

request_refund
  "I want a refund"
  "Please refund this order"

refund_status
  "Where is my refund?"
  "My refund hasn't arrived"
  "How long does a refund take?"

Four layers of evaluation

Intent accuracy is layer one of four. The business only cares about layer four.

Getting the label right is not the goal. The goal is that the user's problem gets solved. Those come apart more often than you would expect, so evaluate each stage separately — otherwise a failure at any layer looks identical from the outside.

Layer 1  Intent recognition
Layer 2  Entity / slot extraction
Layer 3  Tool / workflow selection
Layer 4  End-to-end task success

Take one request — "Change the delivery address for order A123 to the Melbourne office" — and check all four:

Layer 4 — task success rate, the share of requests that ended with the user's problem solved — is the one to lead with when someone senior asks how good the agent is:

Task success rate = requests completed successfully
                    ─────────────────────────────
                          total requests

And it is genuinely possible to fail layer 1 and pass layer 4. If the model labels a query check_order where your ground truth said check_delivery, but both route to the same lookup and the user gets the right answer, the intent metric records a failure that did not exist. That is a good thing to be able to say: it shows you know which numbers are proxies and which one is the outcome.

The metric that matters most: wrong action rate

For any agent that can change data, this outranks accuracy.

A user asks: "If I cancel my order, do I get a refund?"

That is a question about policy. It is not a request to cancel anything. But it contains the word "cancel", it is about their order, and a model under pressure to pick an action will pick cancel_order. If the agent then executes it, the user has just lost their order by asking a question about losing their order.

The full metric set for a serious enterprise agent, by layer:

The last pair is a useful thing to raise unprompted. Clarification is the safety valve for ambiguity — but an agent that asks a question every turn is exhausting, and users abandon it. Tracking unnecessary clarifications separately stops the safe behaviour from quietly becoming the useless behaviour.

Building a test set that isn't flattering

If your test cases are all textbook sentences, your metrics measure nothing.

The temptation is to write the canonical phrasing of each intent and call it a test set:

Cancel my order
Check my order
Request refund

Those will pass, and they tell you nothing, because no real user talks like that. One intent needs many shapes. For cancel_order alone:

Cancel my order.
I don't want this anymore.
Can you stop my order?
I ordered the wrong item.
Please cancel order #1234.
I changed my mind.
Can you undo my purchase?

Then deliberately include the categories that break things:

• Typos and sloppy input — "cancle my ordr"
• Spoken register — fragments, no punctuation, filler words
• Long rambling messages where the actual request is one clause in the middle
• Vague phrasing that genuinely should trigger clarification
• Multiple intents in one message
• Context-dependent messages — "yes, do that" only means something given the previous turn
• Negation — the single highest-value category, below

The negation case deserves its own example, because it defeats every keyword-based system and a surprising number of model-based ones:

"I don't want to cancel my order,
 I just want to change the address."

The word cancel is right there. A keyword rule fires cancel_order and destroys the order. The correct label is change_delivery_address. Put a handful of these in your test set on day one — they are the cases that separate a system that looks like it works from one that does.

The whole thing as one line

If you remember nothing else from this page, remember the chain.

User request
   ↓
What does the user want?
   ↓
Intent
   ↓
What information does that need?
   ↓
Slots
   ↓
What should the system do?
   ↓
Workflow / tool
   ↓
Did it actually solve the problem?
   ↓
Task success

Which is why "intent recognition" is a slightly misleading name for the discipline. It is not an isolated classification problem. It is the first link in:

Everything in this page follows from that. Design intents around executable actions, split them only when execution differs, define their boundaries by their neighbours, keep an escape hatch for unknown and ambiguous, and measure the chain end to end rather than only the first link.

Glossary

Every term used above, defined plainly.