Evaluating an AI agent: building and maintaining its test set

Why the test set comes before everything else

An AI agent is not judged in a demo. In a demo, you ask the questions you know it gets right. In production, users ask their own: badly worded, incomplete, off topic, sometimes deliberately tricky. Between the two, you need a measuring instrument. That is the evaluation set: a list of real cases, each with the expected answer and a way to judge whether the agent gave it.

Without that set, every decision is made on impressions. You change an instruction because one answer was disappointing, and you do not know what that change broke elsewhere. You switch models because of a public leaderboard. You open the agent to everyone because "it seems to work". Impressions see spectacular successes, they miss quiet regressions.

Anthropic puts it simply in its documentation on choosing a model: having a good evaluation set is the most important step. We would go further: it is the first serious deliverable of an agent project, before the first version of the agent itself. It defines what "working" means, and it serves throughout the life of the system. It is also one of the production prerequisites of a business POC.

Start from real cases

An evaluation set invented in a meeting looks like what the project team imagines. A set taken from real activity looks like what the agent will face. The difference is considerable.

The best sources of cases are those that already exist:

• the history of tickets or requests sent to the team concerned;
• emails received on a contact address;
• questions employees ask internal experts, noted during a few days of observation;
• the documents actually processed: purchase orders, invoices, contracts, minutes, with their format flaws.

Claude's documentation insists on this: an evaluation set must mirror the real distribution of tasks. If half your requests are about order tracking, half the set should be too. A set that overrepresents interesting cases gives a flattering and false picture.

Real cases contain personal or sensitive data. They are handled accordingly: extraction limited to what is useful, pseudonymisation where possible, storage at the same protection level as the source. An evaluation set does not get passed around by email like an ordinary working file.

This is where the forward deployed model makes sense: the engineer who sits with you sees the real requests go by, knows which ones recur and which ones cause trouble, and builds the set from what they observe.

Expected answers validated by the business

A case with no expected answer is useless. And the expected answer cannot come from the technical team: only the business knows what is right. An engineer can judge that an answer is clear and well formed. They cannot judge that it applies the right refund rule or the right clause of the framework contract.

In practice, each case in the set includes:

• The input. The question or document, as it actually arrives.
• The expected answer. Written by a business expert, or at least the elements it must contain and those it must never contain.
• The source. The document or rule that justifies the answer, when the agent answers from a document base.
• The expected behaviour. Answer, ask for clarification, decline, or hand over to a person.
• The severity of an error. A wording imprecision does not weigh the same as false information given to a customer.

Business validation takes time, and it is time well spent. It often reveals internal disagreements: two experts giving different answers to the same case. Better to find them before go-live than after a complaint. Organising this collaboration between business and technology is part of the AI product management role.

Covering hard cases and out of scope cases

A set made only of easy cases measures easiness. You must deliberately add what trips an agent up. Claude's documentation lists several families of edge cases to include: irrelevant inputs, overly long inputs, ambiguous requests, harmful requests. We break them down as follows:

Hard cases

• ambiguous questions, where the right answer is to ask for clarification;
• badly scanned, incomplete documents in an unusual format;
• cases where two rules seem to apply;
• casual wording, with typos, abbreviations or internal jargon;
• requests that combine several topics in one message.

Out of scope cases

These are the most important and the most forgotten. A support agent for warranties will get questions about billing, politics, the weather. For each, the expected behaviour is clear: do not make things up, say it is not its role, point to the right contact.

One family deserves particular attention: requests that push the agent to produce data that commits the company. A price, a delivery date, an availability, a discount. The set must check that the agent never invents them and relies on the system that computes them. That is the principle set out in our article on why an AI agent must never quote a price.

Finally, add a few hijacking attempts: instructions hidden in a document, requests to reveal its instructions, prompts for forbidden content. The agent must treat them as data, never as orders.

Defining success criteria

"Good answer" is not a criterion. Claude's documentation recommends criteria that are specific, measurable, achievable and relevant to the use, and notes that most applications need to be evaluated on several dimensions at once. Among those it lists: accuracy on the task, consistency across similar questions, relevance, tone, protection of personal data, response time and cost.

For each dimension you keep, decide how it is scored.

Automatic checks

When the expected answer is precise (a category, an amount extracted from an invoice, a yes or no decision), a program compares the agent's answer with the expected one. It is fast, free to replay and unambiguous. Structure as many cases as possible to allow this kind of check.

Scoring by a model

When the answer is free text, another model can judge it against a rubric: does the answer contain the expected elements, respect the tone, include forbidden information? Claude's documentation recommends using a different model from the one generating the answers. This judge must itself be checked, by comparing its scores with a human's on a sample.

Human review

It remains the reference, and it is expensive. Keep it for sensitive cases, subtle criteria, and checking the two previous methods. The same documentation notes that a large number of automatically scored cases usually brings more information than a small number of hand scored ones. The two complement each other.

Finally, set a threshold per criterion before measuring. Setting the threshold after seeing the result amounts to approving whatever you got.

Replaying the set after every change

An agent changes often: instructions tweaked, documents added, splitting modified, a tool connected, the model replaced. Each change can improve one thing and degrade another. The rule is simple: no change goes to production without the full set being replayed and compared with the previous version.

A few practices make replays useful:

• Automate the run. If replaying the set takes half a day of manual work, nobody will do it. It must start with one command.
• Compare case by case. The overall score can stay stable while important cases have flipped. Look at the list of cases going from pass to fail.
• Keep the history. Keeping results for each version helps you understand when a degradation appeared.
• Hold part of the set back. If you tune instructions while always looking at the same cases, you end up optimising for them. Claude's documentation recommends keeping held-out test cases that you do not look at while tuning.

For an agent answering from documents, also replay the retrieval measure on its own: is the right passage found? Anthropic uses this kind of measure in its contextual retrieval work, and it tells you whether a regression comes from retrieval or from writing the answer.

Changing models is the most visible case. Our article on what Claude Opus 5.5 changes shows how the evaluation set is used to decide on a switch.

Monitoring in production

The evaluation set measures the agent on known cases. Production brings new ones. So you need a second mechanism that observes what actually happens.

• Direct signals. Thumbs up or down, requests to be handed to a human, successive rewordings of the same question, abandoned conversations.
• Technical signals. Response time, errors, cost per conversation, rate of "I don't know" answers.
• Business signals. Complaints, corrections made by teams afterwards, reopened tickets.

Every negative signal is a candidate for the evaluation set. A question handled badly in production, once corrected and validated by the business, joins the set. The set grows with usage, and a corrected error cannot come back undetected.

Monitoring must follow the same rules as the rest of the system: protected logs, defined retention, restricted access. That is one of the principles of our responsible AI memo.

Human review of samples

Automatic indicators do not see everything. An answer can be polite, fast, well rated by the user, and wrong. The only way to know is for an expert to read it.

The practice we put in place: at regular intervals, a business expert reviews a random sample of conversations, plus a sample targeted at flagged conversations. They score each answer against the same rubric as the evaluation set. This produces three things:

• a measure of real quality, independent of automatic indicators;
• new cases for the evaluation set;
• a check on the automatic judge, if there is one, by comparing its scores with the expert's.

The pace depends on volume and risk. An agent answering customers on sensitive topics needs more frequent review than an internal document search assistant. What matters is that the review is scheduled, assigned to a named person, and that its findings lead to action.

Keeping the set alive over time

An evaluation set ages like a document corpus. Rules change, products evolve, a procedure is replaced. An expected answer that has become wrong fails an agent that is right, or passes an agent that is wrong.

• An owner. The set belongs to the business, with technical support. Someone is accountable for its accuracy.
• A review at every rule change. When a procedure changes, the affected cases are updated at the same time.
• Versioning. The set evolves through dated versions, so that comparisons over time stay readable.
• A watched balance. As cases from incidents are added, the distribution can drift towards hard cases. Check now and then that it still reflects real usage.

Common mistakes

• Building the set after the agent. You then test what the agent can do, not what you expect of it.
• Letting the technical team write the expected answers. The set then measures conformity to an engineer's idea of the business.
• Forgetting out of scope cases. The agent shines on its topic and invents as soon as you leave it.
• Only looking at the overall score. Serious regressions hide behind a stable average.
• Stopping at go-live. Without monitoring and review, quality drifts and nobody sees it.

Our article on deploying an AI agent in your company places these steps in the full sequence, and our page on designing an AI agent sets out how we work. If you have an agent planned or already live and want to know how to measure it, our AI diagnostic lays out the cases, the criteria and the monitoring set-up.

Sources

• Claude documentation, Define success criteria and build evaluations
• Claude documentation, Choosing the right model
• Anthropic, Introducing Contextual Retrieval

Frequently asked questions

How many cases do we need to start?

There is no universal number. The set must cover the main families of requests in their real proportions, plus hard and out of scope cases. Better to start with a modest but representative set, then grow it from production.

Who on our side needs to spend time on the evaluation set?

One or more business experts, to provide cases and validate expected answers. The technical team structures the set, automates replays and analyses results. Without the experts' time, the set measures nothing useful.

Can answers be scored entirely by an AI?

Partly. A model judges well the criteria it is given precisely, and it lets you score many cases at low cost. It must still be checked through regular human reviews, and sensitive cases remain reviewed by an expert.

Does the evaluation set replace user acceptance testing?

No, it prepares it. The set checks that the agent answers correctly on known cases. Acceptance testing with users checks that it fits their work, that the interface suits them and that handovers work.

What if the agent does not reach the threshold?

First look at which cases fail and why: retrieval, instructions, data, model. Sometimes the answer is to narrow the agent's scope to the families of requests where it is reliable, and hand the rest to a person.