AI agent evaluation: how we measure accuracy in production

AI agent evaluation means measuring the agent on cases where you know the right answer, before every change and again in production. It has three parts: a test set of real, hand-labeled cases that runs on every change, a threshold per task decided before launch with everything below it routed to a person, and in production we count what people had to fix. With invoices that arrive by e-mail, that is how 70 % go through with no human, and the rest goes to a person on purpose.

This is a technical article for CTOs and developers. Below is how each part looks, with config and code, and where it goes wrong.

What does AI agent evaluation measure?

Two different things, at two different times. Mixing them up is the most common mistake we see: a demo that "looked right" becomes the release criterion.

The offline number tells you whether to ship. The online number tells you whether the offline set still looks like reality. When the two drift apart, the test set is out of date.

How do we build the test set?

From real cases, never from examples a developer makes up while writing the prompt. Made-up cases test what you already thought of.

A case in the set is plain data. A test case can look like this:

- id: invoice-multipage-0142
  input: fixtures/invoices/0142.pdf
  expect:
    supplier_id: "12345678"      # matched by company ID, not by name
    total_gross: "1 240.00"
    vat_rates: [20]
    route: auto
- id: invoice-scan-0219
  input: fixtures/invoices/0219.jpg
  expect:
    route: review                # scans go to a person by rule

The format and the values are illustrative; what matters is what a case records. Note the route field: it tests the routing decision itself, not only the extracted values. An agent that extracts well but sends the wrong cases through alone is still broken.

What does a change look like on the test set?

For a client's online marketplace, an LLM sorts listings into categories. We did not ask another model whether the categories looked right. People labeled real listings by hand and code counted the matches. Then we changed the structure of the prompt and ran the same set again:

That is a test-set number, not live accuracy. It tells us the change was worth shipping. Whether production holds up is what the next two sections measure.

The set is small on purpose. Fifty labeled cases are enough to see a change of that size, and small enough that a person can label them and look at every miss. A bigger set comes from production, one corrected case at a time.

How do thresholds and routing to a human work?

Before launch, write down per task which cases the agent may finish alone. Everything else goes to a person. Build the rule from checks code can verify, not from the model's own confidence.

With invoices that arrive by e-mail, the system reads them and writes them into the accounting software. 70 % go through with no human. The rest goes to a person on purpose: a new supplier, a scanned invoice, more than one VAT rate. A simplified, illustrative rule of this kind can look like this:

type Route = 'auto' | 'review'

function route(inv: Extracted, known: SupplierIndex): Route {
  if (!known.has(inv.supplierCompanyId)) return 'review'   // new supplier
  if (inv.source === 'scan') return 'review'
  if (inv.vatRates.length > 1) return 'review'
  if (!totalsAddUp(inv)) return 'review'                   // net + VAT must equal gross
  return 'auto'
}

Each review branch should stand for a case where the test set showed errors, or where an error costs more than a minute of a person's time. Loosen a branch only on test-set numbers, never because the queue got long.

How to decide which steps wait for a person, and how not to drown them in approvals, is in Human in the loop: when should an AI agent wait?.

What should you measure in production?

Not the share of cases the agent "handled". That number is easy to inflate by routing less. Look at what happened after the agent:

• Corrections per field. Of the cases a person reviewed, how many fields did they change, and which ones. A field that keeps getting corrected is the next test case.
• Share routed to a person, and why. Per routing rule. A sudden jump means the input changed: a new supplier template, a new e-mail format.
• Errors found after auto-approval. Cases that went through alone and were wrong anyway. These are the expensive ones. We recommend reviewing a sample of auto-approved cases regularly; it is the only way to see them.

When the corrections stop dropping, the problem is usually in the inputs, not in the model. A new document type or a supplier who changed their template shows up here weeks before anyone complains.

Where does LLM as a judge fit?

Only where grading needs reading. Extracted fields, categories and routing decisions have a right answer, and code checks them. A judge adds cost and noise there. Where the output is free text, such as a reply to a customer, a judge grading against written criteria is useful, with its known biases. How ours works, and what it got wrong, is in LLM as a judge: a reviewer that tries to refute the work.

Is your agent evaluation ready for production?

Go through your setup:

When we would not build all of this

• The agent only drafts and a person reads every draft. A small test set for releases is enough; production numbers come from the edits anyway.
• There are a handful of cases a month. Labeling and routing rules cost more than checking each case by hand.
• Nobody can say what the right answer is. If two experienced people disagree on the label, fix the process definition before measuring the agent.

Want an agent you can measure?

We build AI agents with the test set, routing and production numbers in your repository, as part of AI agents for business. The code, the test set and the data are yours from day one.

We get back to you within two business days with a few questions about your work. Then we agree on what we do first.

If you'd rather talk first, book a 15-minute intro call.

What would your agent's accuracy be on 50 real cases you labeled yourself?

Frequently asked questions

What is AI agent evaluation?

Measuring whether an agent does the job correctly, on cases where you know the right answer. Before release it runs against a fixed test set; in production it is measured by what people had to correct and what the agent handed to a human.

How big should an AI agent test set be?

Big enough to cover the real mix of cases, including the awkward ones. A few dozen hand-labeled real cases already show whether a change helps or hurts. It grows from production: every case a person had to fix becomes a new test case.

What accuracy is good enough for an AI agent?

There is no general number. Decide it per task before launch, from what an error costs. Where an error is cheap and a person sees it anyway, a lower bar is fine. Where it goes straight to a customer or the books, the agent only acts alone on the cases it gets right and routes the rest to a person.

How is this different from LLM as a judge?

LLM as a judge is one way to grade an answer when the grading needs reading and judgment. Evaluation is the whole system around it: the test set, the thresholds, the routing and the production numbers. Where a right answer exists, code checks it and no judge is needed.

Can we trust the confidence score the model reports?

Not on its own. A model can be confidently wrong. Route on checks code can verify (does the total add up, is the supplier known, is the field present) and on what your test set shows, not on what the model says about itself.