The companies that ship AI fast are not the ones with the best prompts. They are the ones that can tell you within an afternoon whether a change made the system better or worse. Evaluation is the capability that compounds; everything else is interchangeable.

Ask an enterprise team how it knows its AI feature is any good and you will usually get one of two answers: “we tried a few examples and they looked right” or “users have not complained much”. Both are gut feel. Gut feel is fine for a hackathon and fatal in production – because the moment you change a prompt, swap a model or your provider quietly updates theirs, you have no way of telling whether you have just improved the system or broken it.

The highest-leverage investment in any serious AI programme is an evaluation harness. It is also the least glamorous – which is precisely why most teams skip it and then move at a snail’s pace ever after, because they dare not touch a system they cannot measure.

The real problem: AI systems degrade silently

Conventional software fails loudly – a test goes red, an exception is thrown. AI systems fail quietly. A prompt tweak that improves 90% of cases can degrade the remaining 10% – with no error message, no crash, no signal at all, until a customer notices. An update at the model provider can shift the behaviour of your entire system overnight. Without evals you discover regressions like these in production, through complaints, weeks too late and with no idea which change caused them.

This risk of silent regressions is why teams without evals freeze. They stop improving the system, because every change is an unmeasured gamble – and a frozen AI system decays steadily while models, data and user needs move on. Missing evals do not just slow you down; they bring you to a complete standstill.

A framework: the three levels of evaluation

A complete harness works at three levels, and most teams have a fragment of one of them at best.

1. Component evals (offline, automated)

A fixed dataset of inputs with known, correct outputs that runs on every change and scores the system automatically. For retrieval: recall against a labelled set. For extraction: field-level accuracy. For classification: precision and recall per class. This is your regression suite – it has to run in minutes and either clear or block every change. Aim for 100–500 cases from real usage, including the difficult and adversarial ones, not just the easy middle.

2. Output quality evals (semi-automated)

For open-ended text generation there is no single right answer, so you score against rubrics: factual grounding, relevance, tone, safety, whether source references are present. To scale, use a stronger model as a judge against an explicit rubric, but calibrate it against human ratings on a sample so that you can trust it – an unchecked LLM judge has biases of its own. What matters is a reproducible score that moves when quality moves.

3. Monitoring in production (online)

Offline evals cannot anticipate everything real users do. In production you measure refusal rate, fallback rate, latency, cost per call and user signals (corrections, thumbs, escalations). Feed samples of real traffic back into your offline set on an ongoing basis, so that your eval data reflects reality rather than your assumptions at go-live.

A worked example: an extraction system during a model migration

A bank ran an AI system that extracted 14 fields from trade finance documents, and its model provider announced that the version in use was being deprecated – it had to migrate or lose support. Without evals that is a frightening project with no clear boundaries: swap the engine under a production system and hope.

Because we had built an eval set of 220 documents with field-level ground truth, the migration was an afternoon’s work rather than a quarter’s:

• Run the current system: 94.1% field-level accuracy (the known baseline).
• Run three candidate models against the same set: 91.8%, 94.6%, 95.2%.
• Chose the model at 95.2%, confirmed that the two fields the 91.8% candidate degraded were not relevant, and shipped – with evidence instead of hope.

The eval set turned an existential migration into a measured decision. The same harness later caught a prompt “improvement” that raised average quality but cut accuracy on amount fields from 97% to 89% – a change that would have shipped on gut feel and caused real financial errors.

That second catch is the actual return. The harness does not just make you faster; it prevents the confident, well-meant change that quietly breaks exactly the cases that matter most.

How to build the first eval set without boiling the ocean

The objection we hear most often: building an eval set is a huge labelling project nobody has time for. It is not, if you scope it properly. You do not need ten thousand labelled cases; you need a hundred that are representative and difficult. Start by pulling a few weeks of real inputs from production (or the pilot) and deliberately form three pools: the common cases (so that you protect the bulk of the traffic), the known failures (every complaint, every escalation, every “the AI got that wrong” the team can remember) and the edge cases (unusual but legitimate inputs). A hundred cases assembled that way catch more regressions than a thousand random ones, because random samples are dominated by the easy middle, which never breaks.

The labelling does not have to be perfect either. For a first harness it is enough for a single subject matter expert to spend a day establishing the ground truth for a hundred cases so that changes can be cleared; you refine contested labels as they come up. The harness is a living asset: every production error you find becomes a new case, so the set grows precisely in the direction of the inputs that matter. Within a few months the eval set holds more institutional knowledge about what “good” means for your use case than any specification document – and unlike the document, it can be executed.

Another practice that pays for itself: version the eval set alongside the system and record the scores of every release. When somebody asks six months later “did this get better or worse?”, you have a chart instead of a debate. And when a regulator or an internal risk committee asks how you assure quality, “here is our versioned eval suite and the score history of every release” is an answer that ends the conversation. The harness is simultaneously your development accelerator and your compliance evidence – the same artefact serves both, and that is exactly why it is the highest-return investment in the whole programme.

Pitfalls

• An eval set made of easy cases. If it does not contain the difficult, ambiguous and adversarial inputs, a passing score means nothing. Curate from real failures.
• A single global accuracy number. Break it down by field, class or query type – an average hides the regression in the 10% that matters.
• Blindly trusting an LLM judge. Calibrate it against human ratings, or you are measuring the judge’s biases.
• A static eval set. User behaviour shifts; feed real production traffic back into the set, or it goes stale.
• Treating evals as a one-off deliverable. The harness is a living asset that grows with every newly discovered failure mode.

Decision checklist

1. Can we tell automatically, and within minutes, whether a change improved or degraded the system?
2. Does our eval set contain difficult, ambiguous and adversarial cases from real usage?
3. Do we score by segment (field/class/query type) rather than with a single global number?
4. Do we score open-ended outputs against an explicit rubric, with a calibrated judge?
5. Do we monitor refusals, fallbacks, latency and cost in production?
6. Do we feed real traffic back into the eval set to keep it current?
7. Could we handle a forced model migration in an afternoon rather than a quarter?

The gridz view

Prompts are copied in a day, and models become a commodity by the quarter; the evaluation harness is the asset you own and that compounds. It lets you adopt a better model in the week it appears, evidence a quality claim to a regulator and improve continuously instead of freezing out of fear. We build the harness before we scale the feature, because the team that can measure quality is the team that moves fast without breaking the cases that matter. If your AI roadmap has no line item for evals, that is the first thing we would fix – talk to us.

Concretely, the cheapest and most valuable step you can take this month is this: assemble a hundred-case eval set from your own production failures and common queries, have a subject matter expert label it in a day, and wire it in so that it runs on every change. That one artefact turns your AI development from anxious guesswork into measured engineering – you catch the well-meant change that quietly breaks the important cases, and you can adopt the next better model with evidence instead of trepidation. Everything else in a mature AI practice gets easier once you can automatically answer the only question that counts: did this change make it better or worse?