An AI feature that delights users and loses money on every request is not a product – it is a subsidy with a marketing budget. The difference between the two comes down to four architecture decisions you make before launch, not to a vendor discount you negotiate afterwards.

At pilot scale, nobody looks at cost per call. Volume is tiny, the bill is a rounding error, and the team is focused on whether the thing works at all. Then it goes live, volume grows, and somebody in finance asks why the AI line on the cloud bill is up 400% quarter on quarter. By then the architecture is set in concrete, and the expensive decisions were made by default long ago.

LLM inference cost is not an operational detail to be optimised later. It is a design parameter that determines the gross margin of your feature, and it is fixed by decisions you take up front. We have re-architected enough “successful” AI features that were quietly unprofitable to know the pattern – and the savings, when you get it right, land regularly at 60–90% with no loss of quality.

The real problem: costs scale with success, and the defaults are expensive

Every other part of an AI feature has fixed or sub-linear costs. Inference scales linearly with usage – the more people love it, the more it costs, forever. That inverts the usual software economics, where marginal cost trends towards zero. If your unit economics are negative, success makes the problem worse, not better.

And on cost, the defaults are the worst thing you could choose. The reflex is to send every request to the most capable (most expensive) model, every time with the full context, with no caching, and with retries on failure. That is how you build the demo correctly and run the service incorrectly. Most enterprise AI features we audit pay 5 to 10 times what they need to for the quality they deliver.

The four decisions that determine your gross margin

1. Model tiering – stop sending everything to the flagship

Not every request needs your most capable model. Classification, routing, simple extraction and short answers run on a smaller model at a fraction of the cost, and often with identical quality for that particular task. Route by difficulty: a cheap model handles the easy majority, the flagship handles the hard minority, and a classifier (itself cheap) decides which is which. On a typical mixed workload, tiering alone cuts costs by 50–70%, because the easy cases dominate the volume.

2. Caching – stop paying twice for the same answer

Enterprise requests repeat far more often than teams expect. Cache identical and semantically similar requests, and cache the static part of long prompts (system instructions, retrieved boilerplate) through your vendor’s prompt caching where it is available. A cache hit rate of 30–50% is normal on support and internal knowledge workloads, and a cache hit costs a fraction of a fresh generation – sometimes almost nothing.

3. Context discipline – stop stuffing the context window

You pay per token, including every token of context you send along. Teams routinely pad the prompt “to be safe” with the top 20 chunks from retrieval, when 5 reranked chunks would answer better and cost a quarter as much. Tight retrieval and reranking are both a quality lever and a cost lever. Trim few-shot examples as soon as the model is reliable. Every token in the prompt is a token on the bill, on every single call.

4. Output limits and streaming – stop paying for rambling

Output tokens often cost several times as much as input tokens. Cap the maximum output length, instruct for concision, and avoid making the model repeat the input. For interactive features, stream, so that users perceive speed without you needing a faster, more expensive model.

A practical example: the unit economics of a support assistant

A SaaS company’s support assistant handled 60,000 requests a month. The launch architecture: flagship model for everything, full retrieved context (around 6k tokens), no caching, generous output. Blended cost ≈ CHF 0.21 per request → around CHF 12,600 a month, growing with usage.

We applied the four decisions without changing the user experience:

• Tiering: a cheap classifier routed 64% of requests (FAQ-like, simple look-ups) to a small model; only complex cases went to the flagship. −58% on the routed share.
• Caching: a 38% semantic cache hit rate on frequent questions, plus prompt caching of the system instructions. Cache hits cost around CHF 0.005 each.
• Context discipline: reranking brought average context down from 6k to 1.8k tokens – and answer accuracy rose by 3 percentage points, because the model was no longer distracted by marginal chunks.
• Output limits: capped and trimmed for concision via the prompt, which cut average output by around 35%.

New blended cost ≈ CHF 0.046 per request → around CHF 2,760 a month. A 78% reduction, with a slight improvement in quality. The feature went from a negative to a comfortably positive gross margin, and the savings now grow with volume instead of the bill.

The detail worth dwelling on: context discipline improved quality and cost at the same time. The cost-optimised architecture was also the more capable one, because tight retrieval helps the model focus. Cost and quality are not always a trade-off; here they pointed in the same direction.

The trap of optimising for today’s prices

Inference prices fall steadily, and that tempts some teams to conclude that cost optimisation is wasted effort – “the model will get cheaper anyway”. That is a comfortable story and a false one, for two reasons. First, your volume almost always grows faster than unit prices fall; a feature that succeeds generates more requests, longer contexts and richer outputs, so total spend rises even as cost per token falls. Second, falling prices reward precisely those teams who built the swappable architecture from the tiering discussion – they capture every price cut by routing to whichever model is cheapest, while a team hard-wired to one model and one tier waits for that specific vendor to cut that specific price.

The deeper point: the four levers – tiering, caching, context discipline, output limits – are not one-off optimisations you apply and forget. They are architectural properties that keep paying out as the landscape shifts. A caching layer pays off more and more as your traffic becomes more repetitive; a routing layer lets you adopt a newly released cheap model in the week it ships; context discipline keeps your costs flat even as you add more sources to retrieval. Leaving them out because “prices are falling anyway” is like skipping indexes on a database because “disks are getting faster” – technically correct, irrelevant to the structural problem.

There is also a margin-transparency argument that lands in the executive committee. An AI feature with unexamined unit economics is a financial blind spot: nobody can say whether the millionth request is profitable. Once you instrument cost per request and gross margin per feature, AI goes from an unbounded line on the cloud bill to a P&L you can steer – with a deliberate decision about which features deserve the flagship model and which should run lean. More than any single optimisation, that transparency is what separates an AI portfolio that scales sustainably from one that becomes a budget emergency the quarter after it succeeds.

Stumbling blocks

• Modelling costs at pilot volume. Extrapolate to realistic scale; linear costs punish optimism.
• Flagship by default. The single biggest waste. Most requests do not need your best model.
• No caching, because “every request is unique”. Measure before you assume; they rarely are.
• Padding context to be safe. More context means more cost and often worse answers. Rerank and trim.
• Optimising cost without an eval set. You need evals to demonstrate that a cheaper model holds quality – otherwise you are guessing.

Decision checklist

1. What is our cost per request at forecast scale, not at pilot scale?
2. Do we route by difficulty, or do we send everything to the flagship?
3. What is our cache hit rate, and are we caching static prompt parts?
4. How large is our average context, and can reranking shrink it?
5. Are output lengths capped and is concision enforced?
6. Is the feature’s gross margin positive – and does it improve or deteriorate with volume?
7. Do we have evals confirming that cost reductions have not cost us quality?

The gridz view

Treat inference cost as an architecture decision and you can usually cut it by 60–90% at equal or better quality; treat it as a problem for later and success becomes a budget crisis. The four levers – tiering, caching, context discipline, output limits – are unglamorous, and that is exactly where the margin sits. We design AI features to be profitable at scale from day one, with the evaluation harness as the evidence that the cheaper architecture still delivers. An AI feature without a unit economics model is a subsidy waiting to be discovered.

The fastest diagnostic you can run today: take one AI feature in production, work out its actual cost per request at current volume, and extrapolate it to your volume a year from now. If that number makes you uncomfortable, the four levers will almost certainly cut it by more than half without users noticing – and you will have turned a quietly growing line on the cloud bill into a margin you actively steer. Profitability at scale is not a problem you solve after launch; it is an architecture you choose before it – and that choice is yours while it is still cheap. If you would like to run this calculation with your real numbers, talk to us.