Your LLM Bill Is Too High. Here's Exactly Where the Money Goes.

A token-by-token breakdown of what drives LLM costs in production: oversized models, context bloat, silent retries, and caching nobody configured. Plus the levers that cut spend 40-80% without degrading output quality.

The invoice lands and it is two or three times what you modeled. The product didn't change. Traffic is flat. Something in the pipeline is spending tokens you never see, and the dashboard shows one number: total spend.

LLM cost is rarely one big leak. It is four or five small ones that compound: a model bigger than the task needs, a context window you refill on every call, retries that fire silently, reasoning turned up past the point of return, and caching that was never switched on. Each one is measurable. Here is where to look, and roughly what each costs.

Output tokens are where the meter runs fastest

On most providers, output tokens cost four to five times what input tokens do. A request that reads 2,000 tokens and writes 800 is billed far more for those 800. So the first lever is not "send less" — it is picking the right model for the job and stopping generation the moment the answer is complete.

The common waste is reflex: routing every call to the most capable and most expensive model because it feels like the safe default. Classification, extraction, routing, short rewrites, and format conversion run fine on a small model at a fraction of the price. Reserve the frontier tier for what actually needs it.

• Route by task: a cheap model to classify/extract/route, a mid tier for most generation, the top tier only for hard reasoning or long agentic loops.
• Cap max output tokens per endpoint. With no ceiling, a runaway response can bill 100K tokens before it stops.
• When you do need the big model, honesty pays: it often finishes a hard task in fewer turns than a small model that loops. Cheaper per token is not always cheaper per task.

The context window you refill on every single call

The API is stateless. Every turn resends the entire conversation so far — system prompt, tool definitions, all prior messages. A 20-turn chat doesn't pay for 20 messages; it pays for message 1 twenty times, message 2 nineteen times, and so on. Uncached, the cost of a conversation grows with the square of its length.

The other bloat source is stuffing: RAG pipelines that inject 15 retrieved chunks when 3 would answer the question, system prompts carrying every instruction the product ever needed, tool schemas for 40 tools when the model uses 5. All of it is billed as input on every request.

• Retrieve fewer, better chunks. Rerank and keep the top 3–5; more context often lowers answer quality, not just raises cost.
• Trim tool definitions to what the current step needs; for large tool sets, consider tool search.
• Summarize or compact long histories instead of resending them verbatim.

The caching nobody configured

Prompt caching is the single biggest lever most teams leave off. A cache read costs roughly 10% of the normal input price; a cache write costs about 25% more than a normal read at the short TTL. If a stable prefix — system prompt, tool definitions, a shared document — repeats across requests, caching it pays for itself after the second call.

The catch is that caching is a prefix match. One changed byte anywhere in the prefix invalidates everything after it. A timestamp in the system prompt, a UUID near the top, JSON serialized with keys in non-deterministic order — any of these silently drops your hit rate to zero while you keep paying the write premium.

• Keep stable content first: a frozen system prompt and a deterministic tool order. Put volatile content (timestamps, per-request IDs, the user's actual question) last.
• Verify it works: check the cache-read token count in the usage field. If it stays zero across identical-prefix requests, something upstream is invalidating the cache.
• Serialize JSON with sorted keys. An unsorted dict is a silent cache killer.

Retries you are paying for twice

SDKs retry failed requests automatically — typically twice, with exponential backoff, on rate limits (429) and server errors (5xx). That is good for reliability and invisible on your cost dashboard. But a long-output request that times out, retries, and then succeeds bills you for the full second attempt regardless, and in some setups for the partial work of the first.

The quieter version: a long non-streaming request hits the SDK's HTTP timeout, retries, times out again, and you pay for three full generations to get one answer. Streaming doesn't lower the token price, but it prevents these timeout-driven duplicate runs on long outputs.

Reasoning turned up past the point of return

Extended thinking and higher "effort" settings improve hard tasks by letting the model reason before answering — and those reasoning tokens are billed as output, the expensive kind. On a genuinely hard problem, that is money well spent. On a simple lookup or a classification, it is pure overhead: you pay for a paragraph of deliberation to produce a one-word answer.

The failure mode is a global default — turning maximum effort on for every route because it helped one hard endpoint. Match the reasoning depth to the task: high effort for multi-step reasoning and agentic work, low or off for routine calls. Measure it; high effort often shows diminishing returns well before the maximum setting.

Work that doesn't need to happen in real time

A large share of LLM work is not latency-sensitive: nightly enrichment, back-catalog classification, evals, report generation. Running it through the synchronous API at full price is a habit, not a requirement. Batch endpoints typically process the same requests at half the cost, within a window of a few hours up to a day.

• Move offline jobs — embedding backfills, bulk classification, scheduled summaries — to the batch tier for roughly 50% off.
• Keep interactive, user-facing calls on the synchronous path; batch is for throughput, not response time.

The one habit that finds all of it

You cannot cut what you cannot see. These leaks persist because the invoice reports one total, not a per-request breakdown. Every response carries a usage object — input tokens, output tokens, cache reads, cache writes. Log those four numbers per request, tagged by endpoint and model, and the cost structure stops being a mystery. Most of the fixes above become obvious the moment you can see which calls are expensive and why.

• Log per-request token usage — input, output, cache-read, cache-write — tagged by route and model.
• Ask of each endpoint: right-sized model? capped output? cached prefix? bounded context? matched reasoning depth? batchable?
• Re-check after every prompt change. A new timestamp or a reordered field can quietly kill a cache that was saving you 90%.