Model latency explained
Include time to first response.
← Back to Compute College
Compute College
Tokens per second explained
Learn what tokens per second means, how model throughput affects AI applications, and why throughput matters for AI compute capacity planning.
Plain-English definition
Tokens per second is the rate at which a model generates output tokens after response generation begins, making it a useful throughput measure for model serving.
Memory trick: Tokens per second is the model output speedometer.
Why it matters
Output throughput influences user wait time and how much demand a serving stack can handle. Faster useful output may let the same infrastructure serve more work, although other bottlenecks still matter.
- Output throughput sets user wait time and how much work a serving stack can handle.
- Faster useful output can let the same infrastructure serve more requests.
- But other bottlenecks (first-token, batching) still bound real capacity.
Simple example
At an illustrative 50 generated tokens per second, an output of 1,000 tokens takes about 20 seconds after generation starts, before accounting for queueing or first-token delay.
- At 50 tokens/sec, a 1,000-token output takes about 20 seconds after generation starts.
- That excludes queueing and first-token delay, which users also feel.
- Throughput and latency are related but not the same measurement.
Example figures are illustrative calculations, not current quoted market prices.
Market signal
How to read the market signal
Higher usable throughput can reduce effective serving cost or expand capacity; falling throughput under load can reveal demand pressure on serving systems.
- Higher usable throughput can lower effective serving cost or expand capacity.
- Falling throughput under load can reveal demand pressure on serving systems.
- Throughput per dollar is a better capacity read than raw tokens per second.
Market read: throughput per dollar, not raw tokens per second, is the capacity signal; a drop under load can flag serving-side demand pressure. Figures here are illustrative unless explicitly sourced and dated — see our methodology.
Common mistake
Do not confuse tokens per second with the full user experience: first-token latency, output length, quality, and batching also matter.
Practical takeaway
What you can do with this
Use measured tokens per second alongside expected output length and concurrent demand to estimate response time and required serving capacity.
- Combine measured tokens per second with expected output length to estimate response time.
- Factor concurrent demand to size required serving capacity.
- Compare throughput per dollar across candidate models.
Decision check: have you combined tokens per second with output length and concurrency to size capacity, rather than quoting peak throughput alone?
Compute College
Follow model releases as market signals
Follow model releases as AI compute market signals in the ComputeTape Morning Brief.
Compute College track
Model Benchmarks & AI Compute Economics
Step 8 of 23: Tokens per second explained