Every decision about deploying AI comes down to a question teams rarely ask out loud: what does a wrong answer cost, and who pays for it? The answer should determine your automation threshold – not your ambition and not the model’s average accuracy.

The metric everyone chases is the automation rate – the share of cases AI handles without a human. That is the wrong target. Optimise for the automation rate and you will automate cases you should not, because that metric treats every error as equally cheap. It is not. The cost of a wrong answer ranges from “trivial, the user shrugs” to “breach of regulatory requirements” or “wrong dosage” – and good design is built entirely around that asymmetry.

We have designed the human-in-the-loop for systems where a mistake is embarrassing, and for systems where a mistake ends up in court. The principle is the same: design around the cost of an error, then automate as aggressively as that cost allows – no more and no less.

The real problem: average accuracy hides the cases that matter

“The model is 94% accurate” is a meaningless basis for an automation decision, for two reasons. First, the 6% are not randomly distributed – they cluster in the difficult, unusual and often consequential cases, exactly the ones where you can least afford errors. Second, accuracy says nothing about what the errors cost. A system that is 94% accurate can be fully automated in complete safety (errors are cheap and get caught downstream) – or automating it at all would be negligent (the 6% contain six-figure mistakes). The number on its own does not tell you which of the two you are looking at.

What you need is not higher accuracy but a map of where errors land and what they cost – and a mechanism that routes the cases that turn expensive when they go wrong to a human.

A framework: confidence × consequence

Two dimensions decide how each case is handled:

• Model confidence – a calibrated estimate of how likely this particular output is to be correct. “Calibrated” is the hard part: the score has to genuinely mean what it says, validated against your eval set – and not be a raw softmax value that you hope correlates with correctness.
• Consequence – what a wrong action at this point costs: financially, in regulatory terms, in safety terms, in reputation – and how reversible it is.

Cross the two and you get four treatment modes:

1. High confidence, low consequence → automate. Let it run unsupervised; take samples for quality control.
2. Low confidence, low consequence → automate with easy correction. Act, but make it trivially easy for the user to correct or undo the result. Cheap errors, cheap fixes.
3. High confidence, high consequence → a human confirms. The AI does all the preparation; a human triggers the action with full context. Fast, but with clear accountability.
4. Low confidence, high consequence → the human leads, the AI assists. The human owns the case; the AI supplies information and a recommendation. Never automate this quadrant.

The craft lies in setting the confidence thresholds that move cases between the modes – and that is an economic calculation, not a gut call.

Set the threshold by the cost of errors

The automation threshold should sit where the expected cost of an automated error equals the cost of a human review. Below that confidence, a human is cheaper than the error; above it, the review is wasted effort. Concretely, you need three numbers: the cost of a wrong automated action (C_wrong), the cost of a human review (C_review) and the calibrated error rate at a given confidence level. Automate when (error rate × C_wrong) < C_review. That turns “how much confidence should we demand?” from a discussion into arithmetic.

A worked example: automated invoice approval

A finance team wanted to automate invoice approval. The naive plan: automatically approve everything the model scored as matching a purchase order. We designed by consequence instead.

• C_review ≈ CHF 4 (a few minutes of an accounts payable clerk’s time).
• C_wrong varied enormously with the amount: a wrong approval of CHF 200 is a small, recoverable mistake; a wrong approval of CHF 80,000 is a serious incident.

So the threshold was made amount-dependent. Low-value invoices (< CHF 1,000) with high matching confidence: approved automatically – the cost of an error is low and recoverable, and reviewing would cost more than the errors do. High-value invoices: a human confirms, regardless of confidence, because C_wrong exceeds C_review many times over. The middle band: automated only above a calibrated confidence at which the expected cost of errors fell below CHF 4.

Result: 71% of invoices approved automatically by count (the long tail of small amounts), but only 22% by value – a human stayed on the trigger for every large payment. Processing time fell sharply where it was safe for it to fall, and the team’s risk exposure barely moved. A design along the lines of “automate everything above 90% confidence” would have approved large invoices on the basis of a number that says nothing about what they cost.

By count that looks like massive automation; by value it is conservative exactly where it needs to be. That split – automate the many cheap cases, safeguard the few expensive ones – is what designing along consequences buys you.

The loop has to tighten over time, not stay fixed

A human-in-the-loop design is not a setting you choose once. It is a dial you turn as evidence accumulates – and the systems that deliver the most value are the ones that start conservative and earn more automation. At launch, with no empirical track record, set the thresholds cautiously – more cases for humans, more confirmation steps. As eval data and production history build up, you see exactly where the model is reliably right and can adjust the threshold there: cases move from “a human confirms” to “automate”, backed by evidence instead of hope. The direction matters: start where errors are cheap to absorb, and extend automation into more consequential territory only once calibrated confidence justifies it.

This is where the feedback loop becomes the engine of value creation rather than just a nice quality measure. Every human correction is a labelled example of where the model was wrong and what the right answer would have looked like – precisely the data that improves both the model and the calibration of its confidence scores. A system built so that human reviews flow back into the eval set gets measurably better at recognising when it does not know something – and that in turn lets you safely automate more. A system that discards those corrections stays at its launch automation rate forever and keeps paying for human reviews on cases the model mastered long ago.

The economics make this concrete: the value of a human-in-the-loop system is not its automation rate today but the gradient at which it can raise that rate safely. Two systems that both start at 40% automation are not equivalent if one is designed to climb to 75% within a year on the basis of accumulated evidence while the other is stuck at 40% because it captures no feedback. When we design systems like this, the path from human correction through the eval set to the recalibrated threshold is a fully fledged part of the architecture – because that loop is exactly what turns a static automation tool into one whose value keeps multiplying.

Pitfalls

• Chasing the automation rate. It rewards automating cases you should not automate. Optimise for safely automated value, not for the automated share.
• Uncalibrated confidence. A confidence score that does not reflect actual accuracy turns the whole framework into noise. Calibrate against evals.
• One threshold for every case. Consequences vary within a workflow; the threshold should too (as the invoice example shows).
• Rubber-stamp reviews. If people approve without really looking, you have the cost of the review and the risk of automation. Design reviews so that they demand genuine engagement, and show why the case was flagged.
• No feedback loop. Human corrections are gold for training and evals – capture them, or you will learn nothing.

Decision checklist

1. For each case type, what does a wrong automated action cost, and is it reversible?
2. Is our confidence score calibrated against actual accuracy?
3. Have we set the thresholds economically: error rate × cost of error vs cost of review?
4. Does the threshold vary with consequence within the workflow?
5. For consequential cases, does a named, accountable person trigger the action?
6. Are human reviews designed to be real rather than rubber-stamping?
7. Do we capture human corrections to improve the system and the evals?

The gridz view

The best AI deployments are quietly conservative where it counts and aggressive where it is safe – and they can explain to you in francs why every threshold sits exactly where it sits. The automation rate is a vanity metric; safely automated value is the real one. Design around the cost of an error, calibrate your confidence and let the arithmetic set your thresholds. We build human-in-the-loop systems for environments where some errors are recoverable and others land in the regulator’s inbox – and the design starts from that distinction, not from what we would like to automate. If you want to draw that distinction cleanly for your own processes, get in touch.