Why the Cheapest LLM Can Cost You the Most
A per-token price is the number on the pricing page. It is easy to sort by and easy to put in a slide. It is also easy to be wrong about. The bill you actually pay is cost per finished task, and the space between those two numbers is where a lot of AI budgets quietly leak. A model that costs a tenth as much per token can still cost you more per answer, and the reasons are mechanical.
Does the cheapest model actually cost the least?
Not reliably. The advertised figure is cost per token. The number that hits your invoice is cost per successful outcome. A more capable model can finish a task in one clean pass on a short prompt. A cheaper one may need a longer prompt, take more than a single attempt, and produce padded output that a downstream step has to clean up. The FinOps Foundation’s working group on token pricing states it directly: “a cheaper, less capable model might require longer, more complex prompts to produce a good result,” with “more retries or generate verbose, low-quality outputs that need further refinement.”
Why does a lower cost per token not mean a lower bill?
The bill moves for reasons you can measure, and they stack on top of each other.
Output costs more than input
On most providers, output tokens are priced at a premium over input, often three to five times higher, per the FinOps Foundation. That asymmetry matters because a cheaper model tends to be chattier. It restates the question and wraps a one-line answer in extra preamble. Every added sentence lands on the expensive side of the meter. A prompt that returns 500 tokens of output instead of 2,000 can cost several times less for the identical input.
Cheaper models retry more
A task that a stronger model clears on the first attempt might take a weaker one two passes, or a fallback to a larger model after the cheap one misses. Each attempt bills in full. If one call in four has to be redone or escalated, the effective rate is nowhere near the sticker rate. Our own benchmark work treats this as the whole question. A model downgrade only counts if the task still HOLDS at the lower tier, and a REGRESSION means you pay twice, once for the miss and again for the redo.
Reasoning tokens bill for thinking you never see
Reasoning models generate an internal chain before they answer, and those tokens bill at the output rate even though they never appear in the response. A short, correct answer can sit on top of a several-thousand-token reasoning trace. Turn extended thinking on for a task that did not need it and the meter runs well past what the visible output suggests.
Put numbers on it and the flip is easy to see.
| Per 1,000 finished tasks | Capable model | Cheaper model |
|---|---|---|
| Output price (per 1M tokens) | $15 | $3 |
| Output tokens per attempt | 500 | 1,500 |
| Attempts per success | 1 | 2 |
| Output tokens billed | 500K | 3,000K |
| Output spend | $7.50 | $9.00 |
The cheaper model wins on the sticker by five times and still lands 20% higher on the bill. These numbers are illustrative. The arithmetic is the point, and your real figures will differ by workload.
How do you compare LLM cost honestly?
Measure the finished task, on your own traffic. Run the same real workload through the candidate model, count every token including the failures and the retries, and divide by the tasks that passed. That per-outcome number is the one to compare across tiers. An industry average cannot tell you whether your prompts and your success bar survive the cheaper model. Your own runs can.
This is what the downsize analyzer in TokenJam does, and why its language stays careful. It flags a session where a cheaper same-family model looks like it could have handled the work, and it labels the finding a candidate with “review before switching.” It never says “safe to downgrade,” because the only evidence that counts is your task passing at the lower tier on your data.
You can measure this locally. Install with pipx install tokenjam, run tj onboard to record your sessions, then tj optimize downsize to see which calls have a cheaper same-family candidate and the estimated recoverable on your own traffic.
- Is a cheaper model per token always more expensive overall?
- No. For simple, well-scoped tasks a smaller model can be both cheaper per token and cheaper per outcome. The paradox shows up when the task is hard enough that the cheaper model needs longer prompts, extra attempts, or heavy output to reach an acceptable answer. The way to know is to price the finished task on your own workload rather than trusting the per-token rate.
- Why do output tokens cost more than input tokens?
- Generating output runs the model forward one token at a time, while input can be processed in parallel, so output carries more compute per token. The FinOps Foundation puts the premium at roughly three to five times the input rate on most providers. A verbose model therefore spends on the expensive side of the meter.
- What are reasoning tokens and do I pay for them?
- Reasoning models produce an internal thinking chain before the visible answer. Those tokens are billed at the output rate and do not appear in the response, so a short answer can carry a large hidden token count. If a task does not need step-by-step reasoning, leaving extended thinking off keeps the bill closer to the visible output.
- How do I compare two models' real cost?
- Run the same representative tasks through each, count all tokens including retries and failed attempts, and divide by the number of tasks that succeeded. Install pipx install tokenjam, run tj onboard, and use tj optimize downsize to see per-session cheaper-model candidates measured on your own traffic. Try tj demo first to see the output before instrumenting a real agent.
Install with pipx install tokenjam to see which of your model calls have a cheaper candidate, measured on your own workload.