Benchmarks · coding
ProgramBench v1 Fully Resolved
Share of public ProgramBench v1 tasks where the implementation passes every hidden behavioral test.
53 models · 54 results · updated 2026-11-05 · vals.ai ↗
Leaderboard
Opus 5.5 max
Heavy line: best score at each price. Thin lines join one model's effort levels. Cost uses each model's median subscription plan, or API price with no plan data. 33 more models under All models in settings.
More from this benchmark
46 published measuresScore vs any published measure46 measures: sub-scores, splits, counts
41 more models under All models in settings.
Breakdown
darker is better within each column · click a column to sort| Model | |||||
|---|---|---|---|---|---|
| Opus 5.5 max | 25655.3M | 124.2M | 25469.8M | 185.4M | |
| Sonnet 5.5 max | 26746.1M | 138.9M | 26556.7M | 189.3M | |
| GPT-6 Astra max | 751.2M | 19.6M | 9.2M | 693.9M | |
| GPT-6.1 Sol max | 1004.1M | 30.4M | 18.1M | 950.1M | 54.0M |
| Gemini 4 Argon high | 5363.0M | 40.8M | 18.7M | 4930.4M | 0 |
| Muse Spark 1.3 max | 10898.5M | 59.1M | 24.7M | 2731.6M | 0 |
| GPT-6 Sol max | 6137.6M | 31.9M | 13.9M | 6073.5M | 63.2M |
| Kimi K3 max | 22310.4M | 87.2M | 54.3M | ||
| GLM 5.3 max | 15021.5M | 68.4M | 45.5M | 14859.3M | 0 |
| Gemini 3.8 Flash high | 8981.1M | 33.8M | 15.1M | 8555.7M | 0 |
| Grok 4.6 high | 2164.9M | 62.7M | 44.3M | 902.6M | 0 |
| DeepSeek V4.1 Flash max | 14371.7M | 70.1M | 46.7M | 14340.4M | 0 |
| GPT-5.6 Terra max | 3001.7M | 22.2M | 9.5M | 2888.8M | |
| GPT-6 Luna max | 1373.3M | 27.1M | 16.7M | 1324.5M | 47.9M |
| Grok 4.7 xhigh | 4016.8M | 40.4M | 21.7M | 1901.6M | 0 |
| Inkling 2 Small max | 50784.2M | 120.5M | 50607.8M | 0 | |
| MiMo V2.6 Flash high | 15388.9M | 107.7M | 80.1M | 15236.8M | 0 |
| MiMo V2.6 Pro max | 8121.2M | 81.7M | 62.8M | 8016.9M | 0 |
| Muse Spark 1.2 xhigh | 2012.7M | 28.0M | 14.1M | 1931.2M | 0 |
| DeepSeek V4 Flash 0731 high | 11633.5M | 60.4M | 37.8M | 11608.8M | 0 |
| DeepSeek V4 Pro 0813 max | 11180.5M | 36.7M | 17.1M | 11150.3M | 0 |
| Gemini 3.7 Flash high | 5272.5M | 33.0M | 9.6M | 5006.3M | 0 |
| GLM 5.3 Flash max | 8901.5M | 41.7M | 22.3M | 8651.9M | 0 |
| Hy4 preview high | 8424.2M | 55.4M | 38.2M | 5827.5M | 0 |
| Inkling max | 4102.8M | 18.6M | 4065.5M | 0 |
All results
54 rows| # | Model | Effort | Score | Subscription $ / task | API $ / task | Harness |
|---|---|---|---|---|---|---|
| 1 | Opus 5.5 | max | 18.5% | $1.03 | $68 | mini-SWE-agent |
| 2 | Fable 5.1 | max | 7.0% | $9.12 | $59 | mini-SWE-agent |
| 3 | Sonnet 5.5 | max | 6.5% | $0.54 | $36 | mini-SWE-agent |
| 4 | GPT-6 Astra | max | 5.5% | $0.50 | $12 | mini-SWE-agent |
| 5 | Opus 5 | max | 3.0% | $1.04 | $60 | mini-SWE-agent |
| 6 | GPT-6.1 Sol | max | 3.0% | $0.17 | $2.87 | mini-SWE-agent |
| 7 | Haiku 5.5 | max | 2.5% | $0.14 | $6.83 | mini-SWE-agent |
| 8 | Gemini 4 Argon | high | 2.5% | $13 | $18 | mini-SWE-agent |
| 9 | Muse Spark 1.3 | max | 2.5% | $27 | $54 | mini-SWE-agent |
| 10 | Fable 5 | max | 2.0% | $8.53 | $76 | mini-SWE-agent |
| 11 | GPT-6 Sol | max | 2.0% | $0.36 | $12 | mini-SWE-agent |
| 12 | Kimi K3 | max | 2.0% | $6.41 | $70 | mini-SWE-agent |
| 13 | GLM 5.3 | max | 1.5% | $2.28 | $22 | mini-SWE-agent |
| 14 | GPT-5.6 Sol | max | 1.5% | $0.42 | $15 | mini-SWE-agent |
| 15 | Opus 4.8 | max | 1.0% | $0.54 | $31 | mini-SWE-agent |
| 16 | Gemini 3.8 Flash | high | 1.0% | $0.44 | $11 | mini-SWE-agent |
| 17 | Grok 4.6 | high | 1.0% | $1.72 | $17 | mini-SWE-agent |
| 18 | Sonnet 4.6 | max | 0.5% | mini-SWE-agent | ||
| 19 | DeepSeek V4.1 Flash | max | 0.5% | $0.064 | $0.90 | mini-SWE-agent |
| 20 | GLM 5.2 | max | 0.5% | $0.37 | $13 | mini-SWE-agent |
| 21 | GPT-5.5 | xhigh | 0.5% | $0.19 | $6.95 | mini-SWE-agent |
| 22 | GPT-5.6 Terra | max | 0.5% | $0.15 | $5.38 | mini-SWE-agent |
| 23 | GPT-6 Luna | max | 0.5% | $0.013 | $0.18 | mini-SWE-agent |
| 24 | Grok 4.7 | xhigh | 0.5% | $4.32 | $46 | mini-SWE-agent |
| 25 | Inkling 2 Small | max | 0.5% | $8.46 | $8.46 | mini-SWE-agent |
| 26 | Inkling Small | max | 0.5% | $0.39 | $0.48 | mini-SWE-agent |
| 27 | MiMo V2.6 Flash | high | 0.5% | free | $0.47 | mini-SWE-agent |
| 28 | MiMo V2.6 Pro | max | 0.5% | $0.61 | $0.73 | mini-SWE-agent |
| 29 | Muse Spark 1.2 | xhigh | 0.5% | $1.23 | $2.47 | mini-SWE-agent |
| 30 | Muse Spark 1.3 | xhigh | 0.5% | $5.78 | $12 | mini-SWE-agent |