#agents
27 posts
-
Evals vs Benchmarks vs Certification: What Each One Actually Proves
A mechanism-level explainer of what an eval, a benchmark, and per-decision certification each prove about an AI agent or a model change, why an aggregate pass rate is not per-decision safety, and where the hard, largely-unsolved part still lives.
-
Some of Your Agent's Tasks Don't Need an Agent
Parts of your agent run the same deterministic tool-call sequence on every run, and you pay model tokens each time to reproduce what a plain script would do for free.
-
Stop Paying to Re-Plan Work Your Agent Already Solved
Agents re-derive the same plan skeleton on every run. TokenJam clusters your runs by plan shape, isolates the planning tokens, and exports the repeated plans as templates you can feed back in.
-
Instrument Your AI Agent, Then Find Where the Money Goes
Patch your provider client in one line so the TokenJam SDK captures every LLM call to a local, on-disk trace, then run local analyzers that turn those traces into priced savings across your self-built agent.
-
CLAUDE.md Best Practices: What a Good One Actually Looks Like
A good CLAUDE.md gives Claude Code the architecture, the critical rules, and the worktree discipline it needs to work in a multi-agent repo. Here's the anatomy, with real examples.
-
Why Model Autorouting Savings Need a Proof Step
Model autorouting to a cheaper, smaller, or open-source model shows a big savings number before any work is redone. That figure is a prediction of your AI spend, not a result. Here's why LLM cost savings from an autorouted swap stay a hypothesis until you replay it on your own tasks and measure whether quality holds or regresses.
-
What Actually Costs Money in an Agent Loop
A mechanism-level breakdown of where tokens get spent every turn an agent runs: input, output, cache reads vs cache writes, context bloat, tool overhead, fan-out, and retries.
-
Introducing TokenJam Bench: Benchmarks & Evaluations for Agents and LLMs
TokenJam Bench is an open-source tool to benchmark and evaluate LLMs and agents. Run a candidate model against an original on real, executable task suites and get a measured pass-rate, confidence intervals, and a holds-or-regressed verdict. Local, no signup.
-
What is AI model autorouting?
AI model autorouting picks a different model per request to cut cost without losing quality. How it works, what the research shows, and why measurement comes first.
-
The problem with TokenMaxxing
TokenMaxxing is fun because someone else pays for it. Here's why the subsidy is ending, what Fable 5 just signaled, and how to find your own multiple.
-
What is an agent loop?
Agent loops: the program that prompts your agent for you, checks its own work, and decides when to stop. The lineage from ReAct to orchestration, and why the loop is now the expensive part.
-
Reddit is 40% of your agent's retrieval surface
What 150K LLM citations tell builders about prompt-time grounding, eval coverage, and the source biases their agents inherit by default.
-
Cost dashboards tell you the bill. They don't tell you what to change.
The gap between reporting agent cost and recommending what to do about it. Why an honest recommendation needs to be validated against the user's own data, and the recent research that makes that validation cheap.
-
Where Your AI Agent Bill Goes: 5 Token Waste Patterns
Where your AI agent bill actually goes: the 5 token-waste patterns (context bloat, runaway loops, model overspend, and more) and the research that fixes each.
-
Subsidized AI Is Ending: The Agent Cost Numbers Are Now Real
Uber burned its annual AI budget in 4 months; one team hit $1.3M in 30 days. The real agent-cost numbers, plus the June billing changes that end the subsidy.
-
The 9-layer agent ecosystem map
A unified map of the agent operations ecosystem: nine layers from observability to token economics, the tools at each, where they are converging, and where the gaps remain.
-
What is AI Agent Token Economics?
Agent token economics: understanding where tokens are spent, why agent costs spike unpredictably, and the optimization patterns (model cascading, prompt compression, semantic caching) for reducing spend without losing quality.
-
What is an agent control plane?
Agent control planes: the runtime layer that governs AI agent behavior across a fleet. Policy enforcement, budget caps, audit trails, and how it differs from observability and guardrails.
-
What is human-in-the-loop for AI agents?
HITL for AI agents: when and how to insert human approval, the patterns (pre/post/exception), the tools that exist, and the async-execution problem.
-
What are AI guardrails?
Runtime constraints on what LLMs say and do: input filtering, output filtering, behavioral checks, and structured output enforcement.
-
What are agent environments and sandboxes?
Where AI agents safely act on code, browsers, and machines: the isolation tradeoffs, the major tools, and the link to evaluation.
-
What is Agent Memory and why does it matter?
How AI agents persist state across sessions, why memory is different from RAG, and the open-source projects building this layer.
-
What is agent evaluation?
Agent evaluation: measuring multi-step trajectories, tool use, and open-ended outputs. Why benchmarks alone don't tell you whether an agent works in production.
-
What is an LLM gateway?
LLM gateways unify provider APIs, add fallbacks and caching, and centralize key management: what they do, when you need one, and the tools that exist.
-
What is OpenTelemetry, and why does it matter for AI agents?
OpenTelemetry, OTLP, and the GenAI semantic conventions: how the CNCF observability standard is becoming the lingua franca for AI agent telemetry.
-
What is agent observability?
How AI agent observability works: capturing tool calls, token costs, traces, and behavioral patterns at production scale.
-
Agents 101: Reasoning, Actions & Autonomy
A foundational definition: what AI agents are, how they differ from chatbots and workflows, and the components that make them work.