Benchmarks · coding
DeepSWE v1.1
Pass@1 on 113 original long-horizon software engineering tasks across 91 repos and 5 languages, run with mini-swe-agent in isolated environments (v1.1 fixes dependency drift and flaky tasks).
28 models · 70 results · data as of 2026-11-06 · deepswe.datacurve.ai ↗
Leaderboard
GPT-6 Astra xhigh
Heavy line: best score at each price. Thin lines join one model's effort levels. Cost uses each model's median subscription plan, or API price with no plan data. 20 more models under All models in settings.
More from this benchmark
3 published measuresAgent steps per task70 results · lower is better
20 more models under All models in settings.
Time per task70 results · lower is better
Output tokens per task70 results · lower is better
All results
70 rows| # | Model | Effort | DeepSWE v1.1 | Subscription $ / task | API $ / task | Agent steps | Time | Output tokens | Harness |
|---|---|---|---|---|---|---|---|---|---|
| 1 | GPT-6 Astra | xhigh | 74.1% ±2.9 | $0.19 | $4.43 | 29 | 19 min | 30k | mini-swe-agent |
| 2 | Gemini 3.8 Flash | high | 73.8% ±1.4 | $0.096 | $2.36 | 166 | 11 min | 143k | mini-swe-agent |
| 3 | Opus 5 | max | 73.6% ±3.9 | $0.21 | $12 | 99 | 32 min | 118k | mini-swe-agent |
| 4 | GPT-6 Astra | high | 73.2% ±3.4 | $0.17 | $3.92 | 27 | 17 min | 27k | mini-swe-agent |
| 5 | GPT-6 Astra | max | 73.2% ±0.8 | $0.32 | $7.50 | 28 | 33 min | 61k | mini-swe-agent |
| 6 | Opus 5 | xhigh | 73.2% ±3.1 | $0.16 | $9.07 | 89 | 26 min | 92k | mini-swe-agent |
| 7 | Opus 5 | high | 72.8% ±1.9 | $0.11 | $6.08 | 73 | 19 min | 64k | mini-swe-agent |
| 8 | GPT-6 Astra | medium | 72.8% ±2.6 | $0.13 | $3.08 | 26 | 15 min | 20k | mini-swe-agent |
| 9 | GPT-5.6 Sol | max | 72.7% ±2.8 | $0.23 | $8.39 | 61 | 19 min | 60k | mini-swe-agent |
| 10 | Gemini 3.8 Flash | medium | 71.0% ±2.3 | $0.080 | $1.97 | 147 | 14 min | 125k | mini-swe-agent |
| 11 | GPT-5.6 Sol | xhigh | 70.7% ±0.8 | $0.13 | $4.70 | 44 | 13 min | 41k | mini-swe-agent |
| 12 | Fable 5 | xhigh | 69.9% ±3.2 | $1.51 | $13 | 68 | 24 min | 80k | mini-swe-agent |
| 13 | Fable 5 | max | 69.7% ±4.0 | $2.44 | $22 | 88 | 35 min | 119k | mini-swe-agent |
| 14 | GPT-5.6 Terra | max | 69.6% ±2.6 | $0.14 | $4.95 | 76 | 17 min | 72k | mini-swe-agent |
| 15 | GPT-5.6 Sol | high | 69.4% ±1.4 | $0.096 | $3.47 | 37 | 9.9 min | 28k | mini-swe-agent |
| 16 | GLM 5.3 | max | 69.0% ±3.0 | $0.41 | $3.99 | 124 | 35 min | 80k | mini-swe-agent |
| 17 | Opus 5 | medium | 68.9% ±1.2 | $0.057 | $3.29 | 52 | 13 min | 37k | mini-swe-agent |
| 18 | Fable 5 | high | 68.6% ±1.1 | $1.03 | $9.18 | 59 | 18 min | 57k | mini-swe-agent |
| 19 | Kimi K3 | max | 68.5% ±4.5 | $0.42 | $4.65 | 98 | 1.3 h | 82k | mini-swe-agent |
| 20 | Grok 4.6 | medium | 67.5% ±2.3 | $0.35 | $3.45 | 70 | 15 min | 50k | mini-swe-agent |
| 21 | GPT-5.6 Luna | max | 67.2% ±4.0 | $0.14 | $3.03 | 102 | 19 min | 73k | mini-swe-agent |
| 22 | GPT-5.5 | xhigh | 67.0% ±6.5 | $0.20 | $7.23 | 82 | 30 min | 46k | mini-swe-agent |
| 23 | GPT-6 Astra | low | 67.0% ±1.3 | $0.068 | $1.60 | 20 | 10 min | 11k | mini-swe-agent |
| 24 | Grok 4.6 | xhigh | 66.7% ±2.2 | $0.57 | $5.50 | 87 | 21 min | 71k | mini-swe-agent |
| 25 | Gemini 3.7 Flash | medium | 65.5% ±3.1 | $0.26 | $2.03 | 117 | 21 min | 94k | mini-swe-agent |
| 26 | Fable 5 | medium | 65.4% ±4.4 | $0.69 | $6.09 | 48 | 14 min | 40k | mini-swe-agent |
| 27 | Gemini 3.7 Flash | high | 65.3% ±1.8 | $0.28 | $2.18 | 125 | 21 min | 107k | mini-swe-agent |
| 28 | Grok 4.6 | high | 65.2% ±1.5 | $0.45 | $4.38 | 79 | 18 min | 61k | mini-swe-agent |
| 29 | GPT-5.5 | high | 64.4% ±3.1 | $0.14 | $5.10 | 62 | 34 min | 31k | mini-swe-agent |
| 30 | GLM 5.3 Flash | max | 63.4% ±4.4 | free | $0.48 | 123 | 26 min | 73k | mini-swe-agent |