Benchmarks · coding
ProgramBench v1 Raw Pass Rate
Average percentage of hidden behavioral tests passed per public ProgramBench v1 task.
53 models · 54 results · updated 2026-11-05 · vals.ai ↗
Leaderboard
Opus 5.5 max
Heavy line: best score at each price. Thin lines join one model's effort levels. Cost uses each model's median subscription plan, or API price with no plan data. 32 more models under All models in settings.
More from this benchmark
48 published measuresScore vs any published measure48 measures: sub-scores, splits, counts
43 more models under All models in settings.
Breakdown
darker is better within each column · click a column to sort| Model | |||||
|---|---|---|---|---|---|
| Opus 5.5 max | 25655.3M | 124.2M | 25469.8M | 185.4M | |
| GPT-6 Astra max | 751.2M | 19.6M | 9.2M | 693.9M | |
| GPT-6.1 Sol max | 1004.1M | 30.4M | 18.1M | 950.1M | 54.0M |
| Gemini 4 Argon high | 5363.0M | 40.8M | 18.7M | 4930.4M | 0 |
| GPT-6 Sol max | 6137.6M | 31.9M | 13.9M | 6073.5M | 63.2M |
| Sonnet 5.5 max | 26746.1M | 138.9M | 26556.7M | 189.3M | |
| Grok 4.7 xhigh | 4016.8M | 40.4M | 21.7M | 1901.6M | 0 |
| Grok 4.6 high | 2164.9M | 62.7M | 44.3M | 902.6M | 0 |
| GPT-5.6 Terra max | 3001.7M | 22.2M | 9.5M | 2888.8M | |
| DeepSeek V4.1 Flash max | 14371.7M | 70.1M | 46.7M | 14340.4M | 0 |
| Gemini 3.8 Flash high | 8981.1M | 33.8M | 15.1M | 8555.7M | 0 |
| Muse Spark 1.3 xhigh | 8950.5M | 45.5M | 19.5M | 8236.6M | 0 |
| MiMo V2.6 Flash high | 15388.9M | 107.7M | 80.1M | 15236.8M | 0 |
| DeepSeek V4 Pro 0813 max | 11180.5M | 36.7M | 17.1M | 11150.3M | 0 |
| MiMo V2.6 Pro max | 8121.2M | 81.7M | 62.8M | 8016.9M | 0 |
| Hy4 preview high | 8424.2M | 55.4M | 38.2M | 5827.5M | 0 |
| Gemini 3.7 Flash high | 5272.5M | 33.0M | 9.6M | 5006.3M | 0 |
| GLM 5.3 max | 15021.5M | 68.4M | 45.5M | 14859.3M | 0 |
| GPT-6 Luna max | 1373.3M | 27.1M | 16.7M | 1324.5M | 47.9M |
| Kimi K3 max | 22310.4M | 87.2M | 54.3M | ||
| Muse Spark 1.2 xhigh | 2012.7M | 28.0M | 14.1M | 1931.2M | 0 |
| DeepSeek V4 Flash 0731 high | 11633.5M | 60.4M | 37.8M | 11608.8M | 0 |
| Inkling 2 Small max | 50784.2M | 120.5M | 50607.8M | 0 | |
| GLM 5.3 Flash max | 8901.5M | 41.7M | 22.3M | 8651.9M | 0 |
| Inkling max | 4102.8M | 18.6M | 4065.5M | 0 |
All results
54 rows| # | Model | Effort | Score | Subscription $ / task | API $ / task | Harness |
|---|---|---|---|---|---|---|
| 1 | Opus 5.5 | max | 87.0% | $1.03 | $68 | mini-SWE-agent |
| 2 | GPT-6 Astra | max | 85.4% | $0.50 | $12 | mini-SWE-agent |
| 3 | GPT-6.1 Sol | max | 85.3% | $0.17 | $2.87 | mini-SWE-agent |
| 4 | Gemini 4 Argon | high | 83.7% | $13 | $18 | mini-SWE-agent |
| 5 | Fable 5.1 | max | 82.7% | $9.12 | $59 | mini-SWE-agent |
| 6 | Opus 5 | max | 82.3% | $1.04 | $60 | mini-SWE-agent |
| 7 | GPT-6 Sol | max | 81.9% | $0.36 | $12 | mini-SWE-agent |
| 8 | Haiku 5.5 | max | 78.7% | $0.14 | $6.83 | mini-SWE-agent |
| 9 | GPT-5.6 Sol | max | 77.6% | $0.42 | $15 | mini-SWE-agent |
| 10 | Fable 5 | max | 76.8% | $8.53 | $76 | mini-SWE-agent |
| 11 | Sonnet 5.5 | max | 76.3% | $0.54 | $36 | mini-SWE-agent |
| 12 | Grok 4.7 | xhigh | 75.7% | $4.32 | $46 | mini-SWE-agent |
| 13 | Grok 4.6 | high | 74.0% | $1.72 | $17 | mini-SWE-agent |
| 14 | GPT-5.6 Terra | max | 72.3% | $0.15 | $5.38 | mini-SWE-agent |
| 15 | Sonnet 5 | max | 72.1% | $0.42 | $24 | mini-SWE-agent |
| 16 | DeepSeek V4.1 Flash | max | 72.0% | $0.064 | $0.90 | mini-SWE-agent |
| 17 | Opus 4.8 | max | 71.9% | $0.54 | $31 | mini-SWE-agent |
| 18 | Gemini 3.8 Flash | high | 71.9% | $0.44 | $11 | mini-SWE-agent |
| 19 | Muse Spark 1.3 | xhigh | 71.7% | $5.78 | $12 | mini-SWE-agent |
| 20 | MiMo V2.6 Flash | high | 71.7% | free | $0.47 | mini-SWE-agent |
| 21 | GPT-5.5 | xhigh | 70.8% | $0.19 | $6.95 | mini-SWE-agent |
| 22 | DeepSeek V4 Pro 0813 | max | 70.1% | $3.41 | $3.41 | mini-SWE-agent |
| 23 | MiMo V2.6 Pro | max | 69.8% | $0.61 | $0.73 | mini-SWE-agent |
| 24 | Hy4 preview | high | 69.2% | $9.66 | $13 | mini-SWE-agent |
| 25 | Muse Spark 1.3 | max | 68.7% | $27 | $54 | mini-SWE-agent |
| 26 | Gemini 3.7 Flash | high | 68.7% | $0.92 | $7.13 | mini-SWE-agent |
| 27 | GPT-5.6 Luna | max | 68.3% | $0.044 | $0.94 | mini-SWE-agent |
| 28 | GLM 5.3 | max | 66.3% | $2.28 | $22 | mini-SWE-agent |
| 29 | GPT-6 Luna | max | 66.2% | $0.013 | $0.18 | mini-SWE-agent |
| 30 | Grok 4.5 | high | 65.2% | $0.65 | $4.12 | mini-SWE-agent |