What is Terminal-Bench?
See terminal-based agent testing.
← Back to Compute College
Compute College
What is an agent benchmark?
Learn what AI agent benchmarks measure and why agentic workflows can drive higher token usage, latency, retries, and AI compute demand.
Plain-English definition
An AI agent benchmark tests whether a system can plan, use tools, take multiple steps, and complete a task rather than return one response to one prompt.
Memory trick: Agents spend compute over steps, not just responses.
Why it matters
Agents are tightly connected to compute economics because one request may generate repeated model calls, large tool results, retries, verification rounds, and long runtime.
- One agent request can generate repeated model calls, tool results, retries, and long runtime.
- That ties agent benchmarks tightly to compute economics.
- A model-only score understates the end-to-end cost of an agent.
Simple example
A coding agent can inspect files, plan a patch, invoke tools, run tests, revise work, and verify an outcome. That chain consumes more serving capacity than a single generated answer.
- A coding agent inspects files, plans, calls tools, runs tests, revises, and verifies.
- That chain consumes far more serving capacity than one generated answer.
- Score reflects task completion; cost reflects the whole multi-step chain.
Example figures are illustrative calculations, not current quoted market prices.
Market signal
How to read the market signal
Improvement on agent tasks can support new long-running workloads, increasing token usage and demand for reliable inference capacity when businesses deploy them.
- Agent-task gains can support new long-running workloads and rising token usage.
- Demand for reliable inference capacity grows when businesses deploy those agents.
- A single-turn read of an agent result understates its capacity draw.
Market read: agent-benchmark gains can open long-running workloads that multiply token usage — a larger capacity draw than single-turn chat. Figures here are illustrative unless explicitly sourced and dated — see our methodology.
Common mistake
Do not price agent workloads as if they were single-turn chat or treat a model-only score as an end-to-end agent cost.
Practical takeaway
What you can do with this
Estimate model calls, tool rounds, input and output tokens, retries, elapsed time, and completion rate before budgeting an agent deployment.
- Estimate model calls, tool rounds, input and output tokens, retries, runtime, and completion rate.
- Budget agents on the full chain, not on chat pricing.
- Validate completion rate before scaling an agent deployment.
Decision check: have you costed the full agent chain (calls, tools, retries, runtime) rather than pricing it as single-turn chat?
Compute College
Follow model releases as market signals
Follow model releases as AI compute market signals in the ComputeTape Morning Brief.
Compute College track
Model Benchmarks & AI Compute Economics
Step 20 of 23: What is an agent benchmark