Tokens per second explained
Measure generation throughput.
← Back to Compute College
Compute College
Model latency explained
Learn what AI model latency means, why it matters for production workloads, and how it connects to model serving cost and infrastructure capacity.
Plain-English definition
Model latency is the time a user or system waits for a usable model response, including time to first token for interactive work and time to complete the full output.
Memory trick: Latency is how long the compute makes the buyer wait.
Why it matters
Latency affects buyer choice and capacity planning. Slower completion can require more concurrent serving capacity or make an otherwise capable model unsuitable for interactive work.
- Latency shapes buyer choice and capacity planning, not just user experience.
- Slower completion can force more concurrent serving capacity for the same throughput.
- A capable but slow model can be unusable for interactive work.
Simple example
A low-priced model that takes much longer to respond may not fit interactive coding assistance or customer support, while a batch workflow may tolerate waiting for a lower bill.
- A cheap, slow model may not fit interactive coding or support.
- A batch workflow can tolerate waiting in exchange for a lower bill.
- Time to first token and full-response time describe different constraints.
Example figures are illustrative calculations, not current quoted market prices.
Market signal
How to read the market signal
Lower latency can increase usage and effective throughput; persistent high latency can signal serving pressure or limit adoption despite attractive benchmark scores.
- Lower latency can lift usage and effective throughput.
- Persistent high latency can signal serving pressure or limit adoption despite good scores.
- Latency trends can reveal capacity strain a price board does not show.
Market read: rising latency under load can signal serving capacity pressure, and can cap adoption even when benchmark scores look strong. Figures here are illustrative unless explicitly sourced and dated — see our methodology.
Common mistake
Do not compare latency without considering output length, task complexity, load, and whether the measurement is first-token or full-response time.
Practical takeaway
What you can do with this
Measure time to first token, total completion time, output volume, completion quality, and cost for each workload class.
- Measure time to first token and total completion time per workload class.
- Pair latency with output volume, completion quality, and cost.
- Match the model to the interactivity the workload actually requires.
Decision check: have you measured first-token and full-response latency under realistic load for this workload, not just average speed?
Compute College
Follow model releases as market signals
Follow model releases as AI compute market signals in the ComputeTape Morning Brief.
Compute College track
Model Benchmarks & AI Compute Economics
Step 7 of 23: Model latency explained