Why output tokens cost more than input tokens
Compare token cost roles.
← Back to Compute College
Compute College
Context window explained
Learn what an AI model context window is and how longer context affects token cost, memory, latency, and model serving economics.
Plain-English definition
A context window is the amount of input and prior output a model can consider in one request or conversation, measured in tokens and sometimes including text, code, or encoded images.
Memory trick: Context window is the model working desk: a bigger desk holds more, but filling it costs more.
Why it matters
Longer usable context can unlock document and codebase workloads, but filling that context increases input token volume, may affect latency, and can raise the cost and memory burden of inference.
- Longer usable context unlocks document and codebase workloads.
- Filling that context raises input-token volume and can raise latency.
- Large inputs also increase the memory burden of inference.
Simple example
A large context allowance can let a team submit an extensive document set in one request. Repeatedly sending large inputs, however, can make the input-token bill much larger than a short-query workflow.
- A large allowance lets a team submit an extensive document set in one request.
- Repeatedly sending large inputs can dwarf the bill of a short-query workflow.
- The advertised window is a ceiling, not a recommendation to fill it.
Example figures are illustrative calculations, not current quoted market prices.
Market signal
How to read the market signal
Long-context improvements matter to AI compute when they cause buyers to run document, research, or agent workloads with more tokens and potentially more serving capacity.
- Long-context gains matter to compute when buyers run more tokens per request.
- Document, research, and agent workloads can scale serving capacity needs.
- A bigger window with no workload behind it is not a demand signal.
Market read: a larger context window signals demand only when buyers actually fill it — more tokens per request, more memory, more serving capacity. Figures here are illustrative unless explicitly sourced and dated — see our methodology.
Common mistake
Do not assume a larger advertised context window is always cheaper, faster, or necessary for your application.
Practical takeaway
What you can do with this
Measure the input your workload actually needs, use retrieval or caching where appropriate, and compare outcome quality against token cost and latency.
- Measure the input your workload genuinely needs.
- Use retrieval or caching instead of resending large inputs where possible.
- Compare outcome quality against the token cost and latency of long context.
Decision check: does your workload truly need the full context window, or can retrieval and caching cut input tokens without hurting quality?
Compute College
Follow model releases as market signals
Follow model releases as AI compute market signals in the ComputeTape Morning Brief.
Compute College track
Model Benchmarks & AI Compute Economics
Step 9 of 23: Context window explained