My cost tool autorouted me to a cheaper model and says I'd save 5×. Can I just switch?

You can, but you'd be acting on a prediction. That 5× is your current rate divided by the cheaper rate over your token volume. It assumes the smaller model produces the same accepted outcome on your tasks, which the calculation never checks. Replay the candidate on a sample of your own work first, read the HOLDS or REGRESSES verdict, and switch if it holds.

Isn't a public benchmark score enough to trust a smaller or open-source model?

It's useful for narrowing the candidate list, not for making the call. A leaderboard measures the model on someone else's tasks. Your prompts, tools, and project context are different, and a model that tops a coding leaderboard can still regress on your specific codebase. The load-bearing test is replaying the candidate on tasks that resemble what you ship.

How is a regression different from just a couple of unlucky failures?

That's exactly what the statistics decide. A confidence interval on each pass rate and a paired test (McNemar for pass/fail results) tell a real drop from sampling noise. In one live run, a math suite dropped 2.5 points with a p-value of 1.000 (noise, a hold) while a code suite dropped 37.5 points at p=0.0001 (a decisive regression). Same swap, two honest verdicts.

Does running the proof cost more LLM cost than autorouting saves?

Rarely, if you keep the sample small. You replay a stratified handful of tasks, not your whole workload, and you control the sample size. The measurement runs to about a dollar on a representative sample, which is trivial next to a workload burning tens or hundreds of dollars a day, and far cheaper than shipping a quality regression you find out about weeks later.

If the cheaper model holds, is it safe forever?

It's safe on the tasks you tested, which is what an honest proof step claims and nothing more. Models get updated, your workload shifts, and new task types appear. That's why the verdict stays scoped to your suite with its interval visible, and why re-running the proof when your work changes beats assuming a past hold still applies.

Why Model Autorouting Savings Need a Proof Step

Why do model autorouting savings need a proof step? Because the number an autorouter shows is a prediction of your future bill, not a result of the work still getting done. Routing to a cheaper model cuts your LLM cost only if that model still produces the outcome you accept on your tasks. No price sheet knows that. The proof step is running the swap on your own workload and measuring whether the quality holds.

You’ve seen the pitch. An autorouter picks a smaller model and shows a downward arrow. A cost dashboard flags a big multiple you could save. A compression pitch promises the same tokens for a fraction of the spend. The number is always clean and always big. And it is always a claim about a swap nobody has actually run.

What is model autorouting savings actually measuring?

Take the arithmetic apart. A “save 4.7×” headline almost always comes from the same calculation: your current model’s per-token rate, divided by a cheaper model’s per-token rate, applied to your recent token volume. Maybe with a token-count adjustment. That’s it.

The calculation is correct as far as it goes. Rate cards are public. Token counts are in your telemetry. Division works. If every one of your tasks came out identical after autorouting to the smaller model, your AI spend would in fact drop by that multiple.

That “if” is doing all the work. The number describes a world where the swap is free of consequences. Your actual world might be that one, or might not. The price sheet can’t tell the two apart, because a rate card has no opinion about whether a lighter open-source model writes the same patch as your current one on your codebase.

Why can’t a price sheet tell you if the LLM cost savings are real?

A saving is real only when the cheaper model does the job you were paying the expensive one to do. That’s an outcome question, and outcomes live in your tasks, not in a rate table.

Predicted vs measured. A confident predicted-savings gauge and arrow on the left, and a measured pass-rate result with a small pass/fail verdict on the right.

Here’s the asymmetry that makes this matter. If the swap holds, you cut spend and lose nothing. If it regresses, you cut spend and the agent quietly gets worse: a subtly wrong refactor, a tool call it skips, an answer that’s close enough to pass review and wrong enough to cost you later. You often don’t notice for weeks. The downside of a bad swap dwarfs the upside of a good one, and the predicted number puts both on the same happy line pointing down.

This isn’t hypothetical caution. We ran a real code suite through a cheaper model and watched 15 tasks that passed on the larger model fail on the smaller one, none going the other way. The predicted savings on that swap looked great. The measured outcome was a 37-point pass-rate drop. A team that trusted the autorouter would have shipped that quietly to save a few dollars a day.

So the honest reading of any autorouted savings figure is: this is what you’d pay if the work still holds. The figure and the “if” are two different claims, and only one of them has been checked.

What is a proof step?

A proof step is replaying the candidate cheaper model on your own workload and measuring whether it produces the outcome you accept, with a real pass criterion and statistics behind the verdict.

Concretely, that means four things:

Your tasks, not a leaderboard. A public benchmark tells you how a model did on someone else’s problems. Your prompts, your tools, your project context are different. The replay runs against work that resembles what you ship.
A real pass criterion. Something a grader can check: the test passes, the tool-call tree matches, a judge model scores the two answers as equivalent. Not “looks fine to me” on a single glance.
A HOLDS-or-REGRESSES verdict, not a vibe. The candidate either kept up on the tasks you ran or it dropped. You get told which.
Statistics, not one run. A single lucky pass proves nothing. A confidence interval on each pass rate and a paired test (McNemar, for pass/fail data) separate a real regression from sampling noise.

The proof loop as a three-stage flow: your real tasks, then a replay of those tasks on the cheaper model, then a verdict that reads HOLDS or REGRESSES.

Both verdicts are the point, and this is the part the autorouted-savings pitch skips. A hold is a genuine win: you now have evidence the swap is safe on your work, and you can bank the lower spend with a clear conscience. A regression is also a win: you found out in a test harness instead of in production, and you avoided trading quality for pennies. The proof step turns a coin flip into a measured decision either way.

How do you actually run the proof?

You need three inputs: the model you run today, the cheaper candidate, and a set of tasks that look like your work. Run both models over the tasks, score each on pass or fail, and read the delta with its confidence interval. If you want the candidate drawn from your real usage rather than a guess, this is where a cost tool that reads your telemetry earns its place: tj optimize flags cheaper-model candidates from the sessions you actually ran, framed as estimated recoverable rather than “saves you,” and the bench (tjb) is the step that replays one of those candidates and reports whether it holds. The optimizer finds the candidate; the bench tests it.

The cost of running the proof is small next to the cost of a bad swap. You’re paying for a sample of replayed tasks, not your whole workload, and you get to set the sample size. A workload burning tens or hundreds of dollars a day can afford a dollar of measurement to avoid shipping a regression.

One honesty note, because it’s easy to overclaim here. A proof step measures the tasks you ran. It doesn’t certify the model on the inputs you didn’t test, and it never should say “certified” or “guaranteed.” The verdict stays scoped to your suite, with its interval visible. Measured on 40 of your tasks means measured on 40 of your tasks. That scoping is a feature: it’s the difference between evidence and marketing.

The takeaway

Every autorouted savings number is a hypothesis wearing the costume of a result. The arithmetic is real. The savings are conditional on a quality claim nobody made, about a swap to a cheaper or open-source model that nobody ran. Before you act on the number, run the swap on your own tasks and read the verdict. Sometimes it holds and you pocket a real cut to your AI spend. Sometimes it regresses and you dodged a quiet quality drop. Either way you traded a guess for a measurement, which is the only trade worth making with your production agents.

Common questions

My cost tool autorouted me to a cheaper model and says I'd save 5×. Can I just switch?: You can, but you'd be acting on a prediction. That 5× is your current rate divided by the cheaper rate over your token volume. It assumes the smaller model produces the same accepted outcome on your tasks, which the calculation never checks. Replay the candidate on a sample of your own work first, read the HOLDS or REGRESSES verdict, and switch if it holds.
Isn't a public benchmark score enough to trust a smaller or open-source model?: It's useful for narrowing the candidate list, not for making the call. A leaderboard measures the model on someone else's tasks. Your prompts, tools, and project context are different, and a model that tops a coding leaderboard can still regress on your specific codebase. The load-bearing test is replaying the candidate on tasks that resemble what you ship.
How is a regression different from just a couple of unlucky failures?: That's exactly what the statistics decide. A confidence interval on each pass rate and a paired test (McNemar for pass/fail results) tell a real drop from sampling noise. In one live run, a math suite dropped 2.5 points with a p-value of 1.000 (noise, a hold) while a code suite dropped 37.5 points at p=0.0001 (a decisive regression). Same swap, two honest verdicts.
Does running the proof cost more LLM cost than autorouting saves?: Rarely, if you keep the sample small. You replay a stratified handful of tasks, not your whole workload, and you control the sample size. The measurement runs to about a dollar on a representative sample, which is trivial next to a workload burning tens or hundreds of dollars a day, and far cheaper than shipping a quality regression you find out about weeks later.
If the cheaper model holds, is it safe forever?: It's safe on the tasks you tested, which is what an honest proof step claims and nothing more. Models get updated, your workload shifts, and new task types appear. That's why the verdict stays scoped to your suite with its interval visible, and why re-running the proof when your work changes beats assuming a past hold still applies.