Benchmarks · safety
SRE Bench
Agent reverse engineering of real-world binary programs without access to source code.
31 models · 31 results · updated 2026-11-05 · vals.ai ↗
Leaderboard
GPT-6 Astra max
Heavy line: best score at each price. Thin lines join one model's effort levels. Cost uses each model's median subscription plan, or API price with no plan data. 20 more models under All models in settings.
More from this benchmark
1 published measuresTime per task37 results · lower is better
21 more models under All models in settings.
All results
31 rows| # | Model | Effort | SRE Bench | Subscription $ / task | API $ / task | Time |
|---|---|---|---|---|---|---|
| 1 | GPT-6 Astra | max | 56.9% | $0.44 | $10 | 18 min |
| 2 | GPT-6.1 Sol | max | 50.8% | $0.16 | $2.69 | 33 min |
| 3 | Gemini 4 Argon | high | 44.3% | $22 | $30 | 1.6 h |
| 4 | Opus 5.5 | max | 33.6% | $0.52 | $35 | 1.6 h |
| 5 | GPT-5.6 Sol | max | 30.5% | $1.17 | $43 | 1.5 h |
| 6 | GPT-6 Sol | max | 30.5% | $0.64 | $21 | 1.1 h |
| 7 | Sonnet 5.5 | max | 30.2% | $0.40 | $27 | 1.9 h |
| 8 | Fable 5.1 | max | 22.9% | $5.02 | $32 | 2.0 h |
| 9 | Opus 5 | max | 12.2% | $0.41 | $24 | 1.3 h |
| 10 | Gemini 3.8 Flash | high | 11.1% | $1.11 | $27 | 2.5 h |
| 11 | GPT-5.6 Terra | max | 9.5% | $0.85 | $31 | 3.3 h |
| 12 | GLM 5.3 | max | 9.2% | $2.85 | $27 | 2.8 h |
| 13 | Haiku 5.5 | max | 5.7% | $0.067 | $3.14 | 58 min |
| 14 | Gemini 3.7 Flash | high | 4.6% | $2.77 | $21 | 1.8 h |
| 15 | MiMo V2.6 Flash | high | 4.2% | free | $0.40 | 3.4 h |
| 16 | GPT-5.5 | xhigh | 3.8% | $0.49 | $18 | 46 min |
| 17 | Grok 4.7 | xhigh | 3.8% | $6.01 | $65 | 1.7 h |
| 18 | Muse Spark 1.3 | max | 3.8% | $13 | $26 | 1.8 h |
| 19 | MiMo V2.6 Pro | max | 3.1% | $0.45 | $0.53 | 2.4 h |
| 20 | GPT-6 Luna | max | 2.7% | $0.11 | $1.55 | 1.4 h |
| 21 | Hy4 preview | high | 2.3% | $3.70 | $4.90 | 1.7 h |
| 22 | DeepSeek V4 Pro 0813 | max | 1.9% | $5.58 | $5.58 | 2.3 h |
| 23 | Inkling 2 Small | max | 1.9% | $6.67 | $6.67 | 2.3 h |
| 24 | GPT-5.6 Luna | max | 1.5% | $0.16 | $3.44 | 1.7 h |
| 25 | DeepSeek V4.1 Flash | max | 0.8% | $0.039 | $0.55 | 1.0 h |
| 26 | GLM 5.3 Flash | max | 0.8% | free | $2.29 | 3.2 h |
| 27 | Grok 4.5 | high | 0.8% | $2.10 | $13 | 1.1 h |
| 28 | Sonnet 5 | max | 0.4% | $0.32 | $18 | 3.9 h |
| 29 | GLM 5.2 | max | 0.0% | $0.79 | $27 | 2.5 h |
| 30 | Inkling | max | 0.0% | $0.20 | $2.23 | 29 min |