SWE-bench repository
Official benchmark code, data, and evaluation documentation.
← Back to Compute College
Compute College
What is SWE-bench?
Learn what SWE-bench measures, why it matters for AI coding agents, and how software-engineering benchmarks connect to AI compute demand.
Plain-English definition
SWE-bench is a software-engineering benchmark that evaluates whether AI systems can resolve real GitHub issues by producing changes to real repositories that satisfy evaluation tests.
Memory trick: SWE-bench is closer to “fix this repo issue” than “write this function.”
Why it matters
Repository-level repair is closer to deployed coding-agent work than short completions. If such workflows become reliable, developers can generate longer, repeated inference demand for debugging, patching, and validation.
- Repository-level repair mirrors deployed coding-agent work more than short completions do.
- Reliable repair workflows generate repeated inference for debugging, patching, and validation.
- That repetition is what turns a benchmark gain into sustained token demand.
Simple example
A task can give an agent a code repository plus an issue description, then evaluate whether the submitted patch resolves the problem under its tests. Tool access and agent scaffold affect both score and cost.
- A task supplies a repo plus an issue and checks whether the submitted patch passes tests.
- Tool access and agent scaffold change both the score and the compute consumed.
- The same model can post very different numbers under different scaffolds.
Example figures are illustrative calculations, not current quoted market prices.
Current example
Primary source
The official SWE-bench repository describes a benchmark for resolving real-world GitHub issues and provides its benchmark variants, including SWE-bench Verified. Last checked: May 24, 2026.
No leaderboard performance claim is made here; consult the official benchmark configuration before comparing systems.
Market signal
How to read the market signal
Read SWE-bench gains as a possible coding-agent demand signal only when evaluation configuration is comparable and the capability is adopted for real engineering work.
- Read SWE-bench gains as a demand signal only when the setup is comparable.
- And only when the capability is actually adopted for real engineering work.
- Subset and scaffold differences can explain a "gain" that is really a setup change.
Market read: a SWE-bench gain signals coding-agent demand only when subset, scaffold, and tools are comparable and the capability is adopted. Figures here are illustrative unless explicitly sourced and dated — see our methodology.
Common mistake
Do not compare SWE-bench values without checking the subset, scaffold, tools, test-time compute, and evaluation date.
Practical takeaway
What you can do with this
Use SWE-bench as capability evidence, then measure your own repository tasks by cost per accepted patch and engineer review burden.
- Confirm the subset (such as Verified), scaffold, tools, and date before comparing values.
- Treat the public number as capability evidence, not a cost estimate.
- Measure cost per accepted patch and reviewer burden on your own issues.
Decision check: do two SWE-bench numbers you are comparing share subset, scaffold, tools, and test-time compute?
Compute College
Follow model releases as market signals
Follow model releases as AI compute market signals in the ComputeTape Morning Brief.
Compute College track
Model Benchmarks & AI Compute Economics
Step 11 of 23: What is swe bench