If my eval passes, is it safe to switch models?

No, not on the eval alone. A passing eval proves the aggregate score held on the cases you ran. It does not tell you which cases failed, whether the failures landed on your important tasks, or whether this specific swap regressed a specific output. A model can hold a high pass rate while getting quietly worse on the slice of work that matters most. Read the eval as necessary evidence and not as a green light, then replay the swap on a sample of your own tasks and check the per-outcome result before you commit.

What's a benchmark vs an eval in plain terms?

An eval scores one system on your chosen cases and reports how it did. A benchmark compares two or more options under one fixed protocol so you can rank them. You run an eval to answer 'how does my agent do on my tasks.' You run a benchmark to answer 'of these models, which ranks highest on this shared suite.' An eval needs no competitor; a benchmark needs at least two and a rulebook they both follow.

Can any tool actually certify that a change is safe?

Not in the strict sense, and be skeptical of anything that claims it. Real certification would prove, per decision, that a specific change did not regress a specific output against ground truth. That needs correct answers for every case, not a sample, which is expensive and often ambiguous to establish. The honest state of the art stops at measuring a change on your tasks with statistics, scoped to what you ran, with the interval visible. When a product says 'certified' or 'guaranteed safe,' ask to see the per-decision, ground-truth check. If it's not there, the word is marketing.

A model tops a public leaderboard. Doesn't that mean it'll work for me?

It means it ranked highest on that suite under that protocol, which is useful for narrowing your candidate list and nothing more. The leaderboard measured someone else's tasks with someone else's grader and settings. Your prompts, tools, and codebase are different, and a model that wins a public coding suite can still regress on your proprietary work. Use the ranking to pick candidates worth testing, then run the load-bearing test: replay the candidate on tasks that look like what you ship.

Why can two models with the same eval score behave completely differently?

Because an aggregate score hides the distribution underneath it. A 95% pass rate can be nineteen easy wins and one hard miss, or the reverse. Two systems can post identical averages while failing on entirely different cases, including the ones you care about most. The single number treats a trivial failure and a critical one the same. To tell the two apart you have to look per case, which is exactly the per-decision view an aggregate eval was built to collapse.

If a benchmark says HOLDS on my tasks, am I done testing forever?

You're done for the tasks you ran, at the time you ran them, which is what an honest verdict claims. Models get updated, your workload drifts, and new task types show up that weren't in your sample. That's why a good verdict stays scoped to your suite with its confidence interval on display, and why re-running the check when your work changes beats assuming a past HOLDS still holds. A scoped, current measurement is worth more than a stale one that overclaimed.

Evals vs Benchmarks vs Certification: What Each One Actually Proves

Three words get used as if they mean the same thing. “We evaled it.” “It tops the benchmark.” “It’s certified.” Each makes a different, load-bearing claim, and the gaps between them are where teams ship regressions. This post takes the mechanism apart. What does each one actually measure, what does it prove, and what does none of them prove.

Three ascending tiers of proof strength labeled EVAL, BENCHMARK, and CERTIFICATION, connected by a rising arrow, each stronger and narrower than the last.

What is the difference between an eval and a benchmark?

An eval measures how your system behaves on a set of cases you chose, and reports an aggregate score. A benchmark compares two or more options under one fixed, shared protocol so you can rank them. Same machinery underneath, different jobs: an eval scores one thing on your cases, a benchmark orders several things under one setup.

The distinction sounds pedantic until you act on it. An eval answers “how does my agent do on my forty tasks.” A benchmark answers “of these three models, which ranks highest on this one standardized suite.” You can run an eval on a single system with no comparison in sight. A benchmark needs at least two contestants and a rulebook they both play by. Confuse the two and you start reading a leaderboard as if it described your workload, which it does not.

What does an eval actually prove?

An eval proves that on the specific cases you ran, the system scored what it scored. That is a real, useful fact. It is also a fact about a sample, not about every input the system will ever see.

Here is the mechanism. You pick cases, run the system, grade each one, and average. The output is a number over that set. The number is honest about the set and silent about everything outside it. If your forty tasks skew toward one kind of prompt, the score reflects that skew. The cases you did not pick are the blind spot, and no aggregate score can see its own blind spot.

There is a second, quieter trap: aggregation hides distribution. A 95% pass rate can mean nineteen easy wins and one hard miss, or it can mean the system nails the trivial cases and fails the ones that matter. The single number treats both the same. Two systems with identical eval scores can fail on completely different cases. That is exactly the situation where an aggregate score feels like proof and functions like a coin flip.

What does a benchmark actually prove?

A benchmark proves relative ordering under one protocol. Model A beat Model B on this suite, with this grader, under these settings. It ranks. It does not certify that the winner works on your problems.

Everything a benchmark tells you is conditional on its setup. The task distribution, the prompts, the scoring rubric, the temperature, the harness. Move any of those and the ranking can shift. This is why the same model can top one public leaderboard and land mid-pack on another. Both numbers are real. They measure different setups. A benchmark is a controlled comparison, and control is the whole value and the whole limit: it holds everything fixed so the comparison is clean, which also means the result only speaks to that fixed world.

For choosing a candidate to investigate, this is genuinely useful. A leaderboard narrows the field. It tells you which models are plausibly in range before you spend anything testing them on your own work. What it can not do is stand in for that test. A model that wins a coding suite built from open-source Python repos may still regress on your proprietary codebase with your tools and your conventions. The benchmark measured someone else’s tasks. Yours are the ones you ship.

What would it take to certify a savings claim?

Certification would prove a specific claim per decision, checked against ground truth: this individual change did not regress this individual output. Not an average over a batch. A verdict on each decision. This is the strongest of the three, and it is the one almost nobody delivers today.

The comparison below is the whole argument in one frame.

A three-column comparison of eval, benchmark, and certification across what each measures, what it proves, and its blind spot. Certification measures each individual decision against ground truth and proves one swap did not regress one output, but per-decision proof is largely unsolved.

Why is per-decision proof so much harder than an aggregate score? Because it needs ground truth for every decision, not a sample. To prove a specific swap did not regress a specific output, you need to know what the correct output was, then show the new output matches it, for that case, individually. An eval sidesteps this by averaging. Certification can not average. It has to be right about each one. Ground truth is expensive to establish and often ambiguous, which is precisely why the honest state of the art stops short of it.

This is why the certification column above ends in “largely unsolved.” It is a category worth naming because it is what people actually want when they ask “is this change safe,” and it is also a frontier, not a shipped feature you can buy. TokenJam does not certify anything, and neither does any tool I would trust that claims to. When a product says “certified” or “guaranteed safe,” read it as marketing until it shows you the per-decision, ground-truth check underneath. That check is the hard problem this whole post circles.

Why isn’t an eval enough before switching models?

Because an eval gives you an aggregate pass rate, and switching a model is a per-decision bet. A 92% eval score does not tell you which 8% broke, whether those were your important cases, or whether this specific decision on this specific input got worse.

On the left, an eval computes one aggregate score over a whole batch of cases. On the right, per-decision verification checks each individual output against a ground-truth reference. The aggregate number and the per-row checks answer different questions.

The asymmetry is what stings. If a model swap holds, you cut spend and lose nothing. If it regresses, you cut spend and the agent quietly gets worse on some slice of work you did not inspect, because the aggregate number stayed high enough to look fine. A regression concentrated in your hardest, highest-value cases can hide behind a pass rate that barely moves. The eval was not wrong. It answered the question it was built for, which was not the question you were really asking.

So an eval is necessary and not sufficient before a swap. It is the cheapest evidence you can gather, and you should gather it. Just do not read “the average held” as “each decision held.” Those are two different claims, and only one of them protects you.

What ships today, and what stays honest about its limits

The strongest thing you can actually run today lives at the benchmark layer, done well. Not a public leaderboard on someone else’s tasks. A benchmark on your tasks, replaying a candidate change and reading whether quality holds with statistics behind the verdict.

That is what tokenjam-bench does, and I want to describe it conservatively. The bench (CLI tjb) takes a change you are considering, replays it against a baseline on tasks that resemble your work, and returns a HOLDS or REGRESSES verdict with confidence intervals and a safety gate. It measures the change. It does not certify it. The verdict is scoped to the tasks you ran, with the interval visible, which is the honest boundary. Measured on forty of your tasks means measured on forty of your tasks, no more.

That boundary is a feature. It is the line between evidence and a claim nobody checked. Any predicted-savings figure, from an autorouter, a cost dashboard, a compression pitch, is arithmetic about a swap nobody has run. The proof step is replaying it on your own workload. We wrote up why a savings number needs that proof step in more depth. TokenJam’s tj optimize surfaces cheaper-model candidates from your own session history, framed as estimated recoverable and never as “saves you,” and the bench is the step that tests one before you switch. The optimizer finds the candidate. The bench measures it. Neither one certifies it, and the product language stays that careful on purpose.

Common questions

If my eval passes, is it safe to switch models?: No, not on the eval alone. A passing eval proves the aggregate score held on the cases you ran. It does not tell you which cases failed, whether the failures landed on your important tasks, or whether this specific swap regressed a specific output. A model can hold a high pass rate while getting quietly worse on the slice of work that matters most. Read the eval as necessary evidence and not as a green light, then replay the swap on a sample of your own tasks and check the per-outcome result before you commit.
What's a benchmark vs an eval in plain terms?: An eval scores one system on your chosen cases and reports how it did. A benchmark compares two or more options under one fixed protocol so you can rank them. You run an eval to answer 'how does my agent do on my tasks.' You run a benchmark to answer 'of these models, which ranks highest on this shared suite.' An eval needs no competitor; a benchmark needs at least two and a rulebook they both follow.
Can any tool actually certify that a change is safe?: Not in the strict sense, and be skeptical of anything that claims it. Real certification would prove, per decision, that a specific change did not regress a specific output against ground truth. That needs correct answers for every case, not a sample, which is expensive and often ambiguous to establish. The honest state of the art stops at measuring a change on your tasks with statistics, scoped to what you ran, with the interval visible. When a product says 'certified' or 'guaranteed safe,' ask to see the per-decision, ground-truth check. If it's not there, the word is marketing.
A model tops a public leaderboard. Doesn't that mean it'll work for me?: It means it ranked highest on that suite under that protocol, which is useful for narrowing your candidate list and nothing more. The leaderboard measured someone else's tasks with someone else's grader and settings. Your prompts, tools, and codebase are different, and a model that wins a public coding suite can still regress on your proprietary work. Use the ranking to pick candidates worth testing, then run the load-bearing test: replay the candidate on tasks that look like what you ship.
Why can two models with the same eval score behave completely differently?: Because an aggregate score hides the distribution underneath it. A 95% pass rate can be nineteen easy wins and one hard miss, or the reverse. Two systems can post identical averages while failing on entirely different cases, including the ones you care about most. The single number treats a trivial failure and a critical one the same. To tell the two apart you have to look per case, which is exactly the per-decision view an aggregate eval was built to collapse.
If a benchmark says HOLDS on my tasks, am I done testing forever?: You're done for the tasks you ran, at the time you ran them, which is what an honest verdict claims. Models get updated, your workload drifts, and new task types show up that weren't in your sample. That's why a good verdict stays scoped to your suite with its confidence interval on display, and why re-running the check when your work changes beats assuming a past HOLDS still holds. A scoped, current measurement is worth more than a stale one that overclaimed.