There is a particular relief in an AI bill that goes up. It means people are using it. Somebody puts it on a slide with an upward arrow and calls it adoption.
Then someone asks what it was spent on, and the room does the thing rooms do.
Most of a mature AI bill is not usage. It is inefficiency. That is no criticism of anyone who has ended up here. The same context re-sent ten thousand times. The flagship model answering questions a far smaller one handles identically. None of it appears on a summary invoice, all of it is fixable, and every fix saves money and carbon in the same proportion.
A climbing bill looks like success. Usually it is not.
The lift company’s revenue tells you how many tickets were sold, not whether anybody got down the mountain. AI spend behaves the same way. It rises when usage rises — and equally when a system prompt quietly gains three paragraphs, when retrieval starts returning more than it needs, when a nightly job goes hourly because that was easier than asking why. None of that is adoption; all of it looks identical on the invoice.
Picture two organisations — hypothetical, not a benchmark — doing identical work to the same standard. One sends everything to its most capable model with the policy manual attached; the other routes the routine majority to a small model with a cached prefix. Same outcomes, wildly different bills. One of them looked.
The question is never “is our AI spend too high?” It is “how much of it bought us something?”
Where the waste actually hides
The flagship model answering beginner questions
A team picks the best available model during the prototype, when the only question is whether the idea works. Then the prototype becomes production and nobody revisits it, because it works and revisiting sounds like a project. So your most expensive model spends its day sorting complaints from enquiries: the helicopter to the nursery slope.
The same context, re-sent forever
Models do not remember, so everything the model needs ships again on every call. That much is the technology. What is not is that nobody ever looks at what is in there: a clarification added after a bad output, three examples, a formatting rule duplicating one six lines above. Nothing is ever removed, and every word is multiplied by every call.
Retrieval that brings the whole filing cabinet
Retrieval is meant to be the efficient option: fetch what the question needs instead of stuffing everything in. A naive implementation chunks badly, ranks loosely and hands over whole documents, because narrowing it down was harder. You asked for the trail map and got the planning application, parking survey included.
No caching, on workloads that repeat all day
Most enterprise AI workloads are extraordinarily repetitive: the same system prompt, the same fifty questions in different words. Reprocessing that from cold buys a fresh lift ticket for every run, and the platforms have already solved this. Anthropic’s prompt caching lets you mark a prefix the model resumes from rather than reprocesses; Azure OpenAI does the same, with one constraint worth knowing: the first 1,024 tokens must be identical, and a single character’s difference is a cache miss. An excellent reason to stop putting the timestamp at the top of your system prompt.
Agent loops that re-reason from scratch
Agents are where cost stops being linear: a call is a call, but an agent is a loop, and every turn carries the accumulated history with it. Give it no memory of what it already established and you have the guide who re-reads the whole trail map aloud before every turn, car park included.
Data taking a scenic route between regions
Underneath sits infrastructure nobody on the AI side thinks about, because it is not an AI problem. Data lands in one region, the model endpoint in another because that is where capacity was in March, the vector store somewhere third. Every request drags data across regional boundaries, and cross-region transfer is billed.
Attribute cost to workloads, or keep guessing
A summary invoice tells you that tokens were processed and what they cost. Not which process consumed them, not who owns it, not whether the same answer was generated forty times before lunch. The snow report that says “winter”: technically accurate, operationally useless.
So the number rises and the only conversation available is blunt: spend more, or use it less. You cannot optimise what you cannot attribute.
The unglamorous fix that pays for everything after it is making the bill legible. Not perfect. Not to the penny. Legible. Every call should carry enough metadata to answer one question: what was this for? Separate deployments, request metadata, resource tags, or at minimum your own logging — the mechanism matters less than the discipline.
Then measure the number that matters, which is not total spend but cost per unit of work. That ratio turns a budget argument into an engineering one, because the expensive workload becomes a named thing rather than a general unease.
Choosing the right model for the run
With attribution in place, model-to-task fit becomes an engineering conversation rather than a theological one. The principle: match the model to the difficulty of the task, not to the importance of the project.
Those two get conflated constantly. Nobody puts a beginner on a black run because the lesson was expensive.
| The task | What it actually needs | What it usually gets |
|---|---|---|
| Sort an email into one of six buckets | A small model and a clear instruction | The flagship, plus a prompt that has been growing since March |
| Reason across contradictory evidence and explain the call | Your most capable model, earning every penny | The same model as everything else |
| Answer the same forty questions all day | A cache, and a small model behind it | A cold, full-price call, every time |
Read down the last column: the problem is not that organisations use models they should not, it is that they use one model for everything. Routing is the fix and it need not be clever — the small model takes the routine path, anything uncertain escalates. That is triage, which every functioning operation already does.
Two structural levers belong here too. Some of what you send a model is not a model problem: lookups, validation and arithmetic are cheaper and more reliable as ordinary code. And work that does not need an answer this second should not pay for one — Microsoft documents Azure OpenAI’s global batch as a 24-hour target turnaround at half the cost of global standard, yet overnight enrichment sits on the real-time path for no reason anybody can name.
Measure quality, or you are just asserting it
Here is where most cost work dies. Somebody proposes a smaller model. Somebody else asks “but will it be as good?” Nobody can answer, because nobody ever measured how good the current one is. The flagship stays, and everyone goes back to work having successfully avoided finding anything out.
“Bigger is better” is not a finding. It is what you say instead of having one.
You need no evaluation platform, just a few dozen real examples from your own traffic, the awkward ones included, and an agreed view of what a correct response looks like. You are not chasing a leaderboard score, only finding out where the cheaper option fails and how badly.
Sometimes it is worse in a way that matters, and you have proved the expensive model earns its keep. Sometimes it fails only at the hard edge, which is what routing is for. Keep the set as the spine of the questions you put to any platform.
Cost per outcome, not cost per token
Cost per token is the number the platform shows you, so it is the number that gets optimised. It is also the wrong one. What the business pays for is a completed piece of work: an invoice correctly coded, a question answered well enough that nobody asks again.
Those two come apart constantly. The cheap ticket that puts you on the wrong side of the mountain is not cheap transport; it is a taxi fare with extra steps.
If your optimisation programme cut cost per token and raised retries, you did not save money. You moved it somewhere nobody is measuring.
This cuts both ways, which is what makes it honest: it argues against reflexively downgrading everything as much as against paying flagship prices for classification. It also settles most bundled-assistant or build-your-own arguments, which are usually held entirely on list price.
Efficiency and sustainability are the same engineering problem
This is the part normally handled by a separate team, in a separate document, with a separate set of adjectives. It is not a metaphor. Inference costs money because it consumes compute. Compute consumes energy. Energy carries a carbon intensity.
There is no trade-off to argue about here. Compute you never needed costs money and emits carbon in the same breath.
That is unusual. Most sustainability conversations ask somebody to accept a cost for a benefit landing somewhere else; this one does not. Right-sizing the model, trimming the prompt, caching the prefix, quietening the loop — each is a cost measure and an emissions measure at once, and the programme you run for the CFO produces the numbers your sustainability reporting needs.
Where to start, if you are starting Monday
Do not begin with a model comparison. It is the interesting bit, and the bit whose results you cannot yet evaluate. Begin by making the bill legible, so spend attaches to a named process with a named owner. Give it a fortnight of real traffic, rank by cost per unit of work, and put the top two or three through the six questions above: model, prompt, retrieval, cache, loop, network.
Build the evaluation set first, so “is it still good enough?” has an answer rather than an opinion. Change one thing. Measure cost per outcome. Then take another lap — the same discipline that makes a first automation worth choosing.
None of this is exotic engineering. It is the ordinary discipline of knowing what you spend and on what, applied to a technology that arrived fast enough that most organisations skipped the step. The bill is not the problem. Not being able to read it is.
