How we keep our AI token bill flat while shipping more

Somewhere in the last two years, "we use AI" quietly became "we have a large and slightly mysterious bill from a model provider." I have watched teams turn a genuine productivity gain into a cloud cost problem, and the frustrating part is that almost all of it is avoidable.

A quick distinction first. The tooling our developers build with is a flat subscription per person, so that bill is already boring. What follows is about the other one: the AI inside the products we build, where usage-based pricing means every design decision is a cost decision. These habits are not sophisticated. They are the engineering equivalent of turning the lights off when you leave a room, and they are the reason the token spend on what we ship stays roughly flat while the amount we ship keeps going up.

1. Ask whether it needs a model at all

The single biggest waste is calling a model for something that did not need one. A lot of "AI features" are a lookup, a rule, or a bit of string handling wearing a trench coat. Before anything else we ask: does this genuinely need judgement, or does it need a function?

When it does need a model, we keep the context small and deliberate. Dumping everything into a prompt "so it has it all" is expensive, slow and, counter-intuitively, worse. Models get distracted by irrelevant material the same way people do.

2. The right model for the job

Not every task needs the frontier model. Renaming a variable across a codebase, writing a migration from a described schema, formatting a JSON fixture: a small, fast model does these perfectly well for a fraction of the cost. We save the big model for the work where judgement actually matters: design decisions, tricky bugs, anything security-adjacent.

Routing by task type is probably half of our total saving, and it costs nothing but a moment's thought at the start of each ticket.

3. Work it out once, then store it

Most of the expensive calls we see in the wild are the same question being asked again and again. Summarise this document. Classify this record. Extract these fields. The input has not changed since the last time, but the feature asks anyway, because nobody stored the answer.

We treat model output like any other computed value: generate it once, keep it next to the thing it describes, regenerate only when the source changes. A queue job that runs on write beats a model call that runs on every read, and the user gets a faster page as well as a smaller bill.

4. Cache what repeats

A lot of what gets sent is the same every time: the system instructions, the schema, the examples. Prompt caching means paying full price for that once and a small fraction after. Structuring prompts so the stable part comes first and the variable part comes last is a five-minute change that pays for itself the same day. The same goes for responses: identical inputs should hit a cache, not a model.

5. Hard stop conditions for agents

Agentic tools are wonderful and they are also capable of looping: try, fail, try a slightly different thing, fail, repeat until the budget or the patience runs out. We give agents an explicit definition of done, a maximum number of attempts, and permission to give up and hand back to a human. An agent that stops and says "I am not confident" is worth far more than one that keeps going.

6. Measure it, per feature, in pounds

Everything above only works if you can see the number. We tag every model call with the feature it belongs to, so the provider dashboard tells you what each part of the product actually costs to run. Two things happen when you do this. The number goes down, because people look at it. And nobody has to guess which feature to fix first.

It also makes the conversation with a client honest. The AI runs on their provider account, the cost is visible per feature, and the design decisions above are the reason it stays small.

What this adds up to

None of these habits make the tools less useful. If anything, they make them more useful, because the constraints force better briefs, clearer tickets and more honest stopping points. The bill stays boring, the output goes up, and nobody has to have an awkward conversation about a line item at the end of the quarter.

If you are running a product and the model spend is climbing faster than the usage, start with numbers one and three. Most of the bill is calls that did not need to happen.