Benchmarks · coding
ProgramBench v1 Almost Resolved
Share of public ProgramBench v1 tasks where the implementation passes at least 95% of hidden behavioral tests.
53 models · 54 results · updated 2026-11-05 · vals.ai ↗
Leaderboard
Opus 5.5 max
Heavy line: best score at each price. Thin lines join one model's effort levels. Cost uses each model's median subscription plan, or API price with no plan data. 33 more models under All models in settings.
More from this benchmark
47 published measuresScore vs any published measure47 measures: sub-scores, splits, counts
43 more models under All models in settings.
Breakdown
darker is better within each column · click a column to sort| Model | |||||
|---|---|---|---|---|---|
| Opus 5.5 max | 25655.3M | 124.2M | 25469.8M | 185.4M | |
| GPT-6 Astra max | 751.2M | 19.6M | 9.2M | 693.9M | |
| Sonnet 5.5 max | 26746.1M | 138.9M | 26556.7M | 189.3M | |
| GPT-6.1 Sol max | 1004.1M | 30.4M | 18.1M | 950.1M | 54.0M |
| Gemini 4 Argon high | 5363.0M | 40.8M | 18.7M | 4930.4M | 0 |
| GPT-6 Sol max | 6137.6M | 31.9M | 13.9M | 6073.5M | 63.2M |
| Grok 4.7 xhigh | 4016.8M | 40.4M | 21.7M | 1901.6M | 0 |
| Grok 4.6 high | 2164.9M | 62.7M | 44.3M | 902.6M | 0 |
| Muse Spark 1.3 max | 10898.5M | 59.1M | 24.7M | 2731.6M | 0 |
| Kimi K3 max | 22310.4M | 87.2M | 54.3M | ||
| MiMo V2.6 Flash high | 15388.9M | 107.7M | 80.1M | 15236.8M | 0 |
| DeepSeek V4.1 Flash max | 14371.7M | 70.1M | 46.7M | 14340.4M | 0 |
| GLM 5.3 max | 15021.5M | 68.4M | 45.5M | 14859.3M | 0 |
| GPT-5.6 Terra max | 3001.7M | 22.2M | 9.5M | 2888.8M | |
| Hy4 preview high | 8424.2M | 55.4M | 38.2M | 5827.5M | 0 |
| Gemini 3.8 Flash high | 8981.1M | 33.8M | 15.1M | 8555.7M | 0 |
| DeepSeek V4 Pro 0813 max | 11180.5M | 36.7M | 17.1M | 11150.3M | 0 |
| MiMo V2.6 Pro max | 8121.2M | 81.7M | 62.8M | 8016.9M | 0 |
| Inkling 2 Small max | 50784.2M | 120.5M | 50607.8M | 0 | |
| Gemini 3.7 Flash high | 5272.5M | 33.0M | 9.6M | 5006.3M | 0 |
| DeepSeek V4 Flash 0731 high | 11633.5M | 60.4M | 37.8M | 11608.8M | 0 |
| GPT-6 Luna max | 1373.3M | 27.1M | 16.7M | 1324.5M | 47.9M |
| GLM 5.3 Flash max | 8901.5M | 41.7M | 22.3M | 8651.9M | 0 |
| Muse Spark 1.2 xhigh | 2012.7M | 28.0M | 14.1M | 1931.2M | 0 |
| Qwen 3.8 27B xhigh | 6949.8M | 41.1M | 29.5M | 0 |
All results
54 rows| # | Model | Effort | Score | Subscription $ / task | API $ / task | Harness |
|---|---|---|---|---|---|---|
| 1 | Opus 5.5 | max | 65.0% | $1.03 | $68 | mini-SWE-agent |
| 2 | Fable 5.1 | max | 53.5% | $9.12 | $59 | mini-SWE-agent |
| 3 | GPT-6 Astra | max | 50.0% | $0.50 | $12 | mini-SWE-agent |
| 4 | Sonnet 5.5 | max | 49.5% | $0.54 | $36 | mini-SWE-agent |
| 5 | GPT-6.1 Sol | max | 46.0% | $0.17 | $2.87 | mini-SWE-agent |
| 6 | Opus 5 | max | 41.5% | $1.04 | $60 | mini-SWE-agent |
| 7 | Gemini 4 Argon | high | 41.0% | $13 | $18 | mini-SWE-agent |
| 8 | Fable 5 | max | 33.0% | $8.53 | $76 | mini-SWE-agent |
| 9 | GPT-6 Sol | max | 32.5% | $0.36 | $12 | mini-SWE-agent |
| 10 | Haiku 5.5 | max | 29.5% | $0.14 | $6.83 | mini-SWE-agent |
| 11 | GPT-5.6 Sol | max | 23.0% | $0.42 | $15 | mini-SWE-agent |
| 12 | Grok 4.7 | xhigh | 21.5% | $4.32 | $46 | mini-SWE-agent |
| 13 | Grok 4.6 | high | 19.0% | $1.72 | $17 | mini-SWE-agent |
| 14 | Muse Spark 1.3 | max | 19.0% | $27 | $54 | mini-SWE-agent |
| 15 | Kimi K3 | max | 17.5% | $6.41 | $70 | mini-SWE-agent |
| 16 | MiMo V2.6 Flash | high | 17.5% | free | $0.47 | mini-SWE-agent |
| 17 | DeepSeek V4.1 Flash | max | 16.5% | $0.064 | $0.90 | mini-SWE-agent |
| 18 | GLM 5.3 | max | 16.5% | $2.28 | $22 | mini-SWE-agent |
| 19 | Opus 4.8 | max | 15.5% | $0.54 | $31 | mini-SWE-agent |
| 20 | GPT-5.6 Terra | max | 15.0% | $0.15 | $5.38 | mini-SWE-agent |
| 21 | Hy4 preview | high | 14.5% | $9.66 | $13 | mini-SWE-agent |
| 22 | Gemini 3.8 Flash | high | 14.0% | $0.44 | $11 | mini-SWE-agent |
| 23 | Sonnet 5 | max | 13.5% | $0.42 | $24 | mini-SWE-agent |
| 24 | DeepSeek V4 Pro 0813 | max | 13.5% | $3.41 | $3.41 | mini-SWE-agent |
| 25 | Muse Spark 1.3 | xhigh | 13.0% | $5.78 | $12 | mini-SWE-agent |
| 26 | MiMo V2.6 Pro | max | 12.5% | $0.61 | $0.73 | mini-SWE-agent |
| 27 | Inkling 2 Small | max | 12.0% | $8.46 | $8.46 | mini-SWE-agent |
| 28 | Qwen 3.8 Max | max | 10.5% | $1.46 | $12 | mini-SWE-agent |
| 29 | Opus 4.7 | max | 9.5% | mini-SWE-agent | ||
| 30 | Gemini 3.7 Flash | high | 9.5% | $0.92 | $7.13 | mini-SWE-agent |