SWE-bench repository
Official benchmark code, data, and evaluation documentation.
← Back to Compute College
Compute College
What is SWE-bench?
Learn what SWE-bench measures, why it matters for AI coding agents, and how software-engineering benchmarks connect to AI compute demand.
Compute & Pricing LessonsLearning path
One concept connected to AI compute market decisions.
5-8 minutesRead time
A practical introduction designed to be completed in one sitting.
SWE-bench / Coding / AgentsTags
Useful for developers, founders, procurement teams, and analysts tracking model-serving economics.
Plain-English definition
Plain-English definition
SWE-bench is a software-engineering benchmark that evaluates whether AI systems can resolve real GitHub issues by producing changes to real repositories that satisfy evaluation tests.
Why it matters
Why it matters
Repository-level repair is closer to deployed coding-agent work than short completions. If such workflows become reliable, developers can generate longer, repeated inference demand for debugging, patching, and validation.
- Capability changes matter economically only when they affect deployed workloads or buyer choices.
- Token volume, latency, retries, and throughput determine how a useful result becomes serving cost.
- A ComputeTape reader should connect model evidence to inference demand and required AI compute capacity.
Simple example
Simple example
A task can give an agent a code repository plus an issue description, then evaluate whether the submitted patch resolves the problem under its tests. Tool access and agent scaffold affect both score and cost.
- Use the example to compare workload economics, not as a current market quote.
- Record the task type, evaluation or workload conditions, and the cost inputs before comparing results.
- A successful result is valuable only if its latency and cost fit the intended production use.
Example figures are illustrative calculations, not current quoted market prices.
Current example
Primary source
The official SWE-bench repository describes a benchmark for resolving real-world GitHub issues and provides its benchmark variants, including SWE-bench Verified. Last checked: May 24, 2026.
No leaderboard performance claim is made here; consult the official benchmark configuration before comparing systems.
Market signal
How to read the market signal
Read SWE-bench gains as a possible coding-agent demand signal only when evaluation configuration is comparable and the capability is adopted for real engineering work.
- Look for adoption, routing, usage-volume, or capacity signals rather than a headline score alone.
- Compare input tokens, output tokens, latency, tool rounds, retries, and completion quality together.
- Keep sourced capability facts separate from interpretation about future AI compute demand.
Market read: this metric becomes an AI compute signal only when it changes serving volume, effective workload cost, or the capacity buyers require.
Common mistake
Common mistake
Do not compare SWE-bench values without checking the subset, scaffold, tools, test-time compute, and evaluation date.
Practical takeaway
What you can do with this
Use SWE-bench as capability evidence, then measure your own repository tasks by cost per accepted patch and engineer review burden.
- Buyers: test the metric on tasks close to the workload you will pay to serve.
- Builders: measure tokens, latency, retries, completion rate, and model price on each test run.
- Analysts: require a source and an adoption mechanism before treating a model result as demand evidence.
Decision check: identify the capability measured, the serving cost driver it affects, and the buyer behavior that would make capacity demand change.
Helpful memory trick
Helpful memory trick
SWE-bench is closer to “fix this repo issue” than “write this function.”
Compute College
Follow model releases as market signals
Follow model releases as AI compute market signals in the ComputeTape Morning Brief.
Compute College track
Model Costs
Continue this Compute College lesson path
Previous lesson
What is a coding benchmark
Continue the Model Costs track.
Next lesson
What is livecodebench
Continue the Model Costs track.