AI model evaluation: how to know if it answers well

AI model evaluation is the part of the project nobody budgets for, and the only one that answers the question that matters: is this good enough to put in front of a customer or a colleague? The scene is always the same. Someone runs a demo, the assistant answers three questions correctly, the room nods and the project moves forward. Six weeks later, in production, it turns out it fails on 20 % of real cases and nobody can say whether that is better or worse than before, because nothing was ever measured.

Playing with it for a while is not evaluation. Evaluation means having a fixed set of cases with the correct answer agreed in advance, running it every time something changes and comparing the numbers with last week's. Without that, every decision — switching vendor, tweaking the prompt, moving to a new model version — is made on gut feel.

This guide explains how to set that up in a normal company with no research team: which cases belong in the test set, which metrics make sense for each task type, how much human review you actually need, and what to demand contractually from whoever is selling you the model or the assistant.

What is AI model evaluation, and why isn't a quick test enough?

Evaluating a model means measuring its behaviour on a set of cases that is representative and stable over time. Three words do the work:

• Representative. The cases come from your real operation, not from tidy examples. If 30 % of the emails hitting customer service are badly written and mix two questions together, that 30 % has to be in the set.
• Stable. The same set, run before and after the change. If you test with different questions every week, you are not measuring the model — you are measuring your own memory.
• With an agreed answer. Someone from the business has decided beforehand which answer is correct. You do not get to decide it after seeing what the model said; that is validating backwards, and it always passes.

The underlying flaw of the demo is that it rewards the easy case. An assistant that handles "what is the standard delivery time?" tells you almost nothing. What decides whether the project lives or dies is what it does with "order 4471 arrived incomplete, will you resend it before Friday or should I cancel?". That question carries context, requires a lookup in a system, and has an answer that costs money if it is invented.

There is also an asymmetry worth being clear about from day one: not all errors cost the same. An answer that is clumsy but correct is a minor problem. A confidently stated but false delivery date is a serious one. Leaking another customer's data is an incident. Evaluation should separate those three levels instead of producing a single accuracy percentage that blends them.

How do you build your own test set?

The test set (what the jargon calls a golden set) is the most durable asset of the whole project. The model will change, the vendor might change; the set stays and lets you compare.

How to build it, in order:

1. Pull 100-200 real cases from history. Tickets, emails, queries, documents. Don't write them yourself: copy them as they arrived, typos and ambiguity included. 2. Stratify. Group by query type and respect the real weight of each group, plus a deliberate quota of hard cases: ambiguous ones, ones with two intents, ones referring to data that doesn't exist, ones out of scope. 3. Write the expected answer. It is written by whoever does that job today, not by the technical team. For classification the label is enough; for drafting tasks, list the points the answer must contain and the ones it must never say. 4. Include trap cases. Questions whose correct answer is "I don't know" or "a person should handle this". These are what catch invention, the most expensive failure mode. 5. Freeze and version it. A file in the repository, dated. If you add cases, that creates version 2 and the report says so.

With 150 well-chosen cases you will catch most problems. With 2,000 badly chosen ones, none. And hold back a handful of cases that nobody uses to tune anything: they are the final control, the thing that stops you optimising the prompt against the exam instead of against the job.

If you can't write the correct answer to a case, the model won't manage either. That case isn't an AI problem: it's an unresolved process problem.

Which metrics should you use for each task type?

There is no universal metric. It depends on what you are asking the system to do. This table covers what shows up in a mid-sized company:

| Task type | Example | What to measure | Reasonable starting threshold | |---|---|---|---| | Classification | Routing an email to a department | Accuracy per class, and above all the confusion matrix | 85-95 % depending on the cost of the error | | Data extraction | Pulling amount and tax ID from an invoice | Accuracy field by field, not document by document | 95 %+ on critical fields | | Answering over documents | Internal search, support assistant | Correct answers, answers grounded in the cited source, invention rate | Invention below 2 % | | Drafting | Draft email or proposal | % of drafts used without heavy editing | Measured against time saved | | Summarisation | Call or meeting summary | Coverage of key points and absence of false facts | Human review by sampling |

Three clarifications that save grief:

• Overall accuracy lies when classes are imbalanced. If 95 % of invoices are correct, a model that always says "correct" scores 95 % and is useless. Look at behaviour on the minority class, which is exactly the one you care about.
• False positives and false negatives don't cancel out. In fraud detection or CV screening, where you set the threshold is a business decision, not a technical one. It belongs to whoever bears the consequence.
• Latency and cost are quality metrics. A model that scores three points higher but takes eight seconds per answer can be a worse product than a simpler one. Measure them in the same table.

For open-ended tasks you can use another model as a judge: give it the question, the expected answer and the system's answer, and have it score. It works reasonably well as a filter and makes continuous evaluation much cheaper, but it has to be calibrated: take 50 cases, score them by hand, compare against the automatic judge and see whether they agree. If the judge passes what your people fail, the judge is broken and what it measures is worthless.

How much human review do you need, and who does it?

All of it. But not always, and not over everything.

The arrangement that works has three layers:

• Baseline evaluation on the test set, with people. Done properly once and repeated whenever there is a relevant change. Two reviewers on the doubtful cases; if they disagree, the process owner decides and that decision gets written into the criteria.
• Continuous automated evaluation on the same set, every time the prompt is touched, the model version changes or the document repository is updated. It should take minutes and run by itself.
• Production sampling, which is the only layer that measures reality. Between 1 % and 5 % of real interactions reviewed by a person each week, with a three-field form: correct?, grounded?, what would you have done?

Who does it matters more than the method. Review belongs to whoever masters the task: the customer service person, the tax specialist, the warehouse lead. The technical team builds the machinery and reads the numbers; it does not decide which answer is good. And that time has to be budgeted: between 8 and 16 hours of a business person for the initial evaluation, and one or two hours of sampling a week afterwards. If nobody has those hours, the project isn't ready for production — and it's better to know that now than later.

This connects with something we covered in why AI tools get abandoned: the team that helps define what a good answer looks like is the team that defends the system when it fails. The one that just receives it finished abandons it after the third odd answer.

What should you demand from a vendor before you pay?

This is where evaluation stops being a technical exercise and becomes a clause. What you can ask for without being unreasonable:

• A test on your data, not their demo. Have them run your test set and deliver results broken down by case type. If they refuse to touch anonymised real cases, that is a bad sign.
• Broken-down results, not a headline. "95 % accuracy" means nothing without knowing on which cases and with what distribution. Ask for the per-category table.
• A measured invention rate. For any system answering over your own documents — see RAG for business — have them state what percentage of answers isn't supported by the cited source, and how they measured it.
• What happens when they change model version. This is the most commonly forgotten point. A version change you didn't witness can move results in either direction. Demand advance notice and a test-set run before it is applied.
• Traceability. A log of what was asked, which sources were used and what was answered, with an agreed retention period. Without it there is no way to investigate an incident or respond to a complaint.
• Ownership of the test set and the results. Yours. That is what lets you hold the next vendor to the same standard.

One negotiating detail: ask for the implementation payment to be tied to clearing agreed thresholds on your test set, not to delivering a demo. It changes the conversation completely, and serious vendors accept it without blinking. The full framework for that conversation is in how to choose an AI consultancy.

How do you keep evaluation alive once the system is in production?

A model doesn't decay on its own, but the world around it changes: new products arrive, regulations shift, the document repository is updated, the vendor ships a new version. What used to work starts failing on the new cases, and nobody notices until the complaint lands.

The minimum maintenance that survives day-to-day reality:

• A panel with four numbers reviewed monthly: accuracy on the test set, invention rate, percentage escalated to a person, and cost per interaction. Nothing more. If the list grows to fifteen metrics nobody reads it — the same mistake we described in executive dashboards.
• Feed the set with real failures. Every production complaint becomes a new test case. It is the cheapest way to keep the set representative.
• A threshold that triggers a review. For example: if accuracy drops more than three points against the baseline, the change is halted and investigated. Written down, with an owner.
• A change log. Every prompt change, version bump or source update documented with its date and its evaluation result beside it. That is part of the MLOps job, and it is what lets you roll back when something gets worse.

Frequently asked questions

How many cases does a test set need?

Between 100 and 200 well-stratified real cases cover most projects in a mid-sized company. Variety and the inclusion of hard cases matter far more than raw volume. A set of 2,000 easy examples measures worse than one of 150 chosen well.

Can I evaluate a model without technical people on the team?

For the part that decides — which answer is correct, which error is unacceptable, where the threshold sits — no technical profile is needed, and it is actually better handled by the business. For automating the test run and measuring latency and cost you do need someone technical, in-house or from the vendor.

What error rate is acceptable?

It depends on the cost of the error and on what you compare it with. The right baseline is not perfection but what happens today without AI: a tired person at the end of the day also makes mistakes. For tasks with human review afterwards you can live with 5-10 % error; for processes that write straight into a system with nobody looking, demand 98 % or better and add controls.

Are public model rankings useful for choosing?

They are useful for ruling options out and for a general sense of the level, but not for deciding. They measure generic tasks that look nothing like your operation, and a model that ranks higher may do worse with your documents and your vocabulary. The only test that decides is yours.

---

If you have an assistant or a model running and you can't state with a number whether it answers well, start with the test set: it is a couple of days of work and it changes every conversation that follows, pricing included. If you'd rather we looked at your specific case together, request the audit or write to us and let's talk: a single conversation usually makes it clear whether what you have can be measured or needs tidying up first.