Benchmarks · agentic
WeirdML v3
Fully agentic: 11 hand-made ML and data-analysis tasks run in coding-agent harnesses, where models must explore unfamiliar data and build pipelines with limited data, unspecified goals or little feedback. Overall effective score (95% interval).
25 models · 25 results · data as of 2026-11-06 · htihle.github.io ↗
Leaderboard
GPT-6 Astra xhigh
Heavy line: best score at each price. Thin lines join one model's effort levels. Cost uses each model's median subscription plan, or API price with no plan data. 12 more models under All models in settings.
More from this benchmark
1 published measuresOutput tokens per task25 results · lower is better
9 more models under All models in settings.
All results
25 rows| # | Model | Effort | WeirdML v3 | Subscription $ / task | API $ / task | Output tokens | Harness |
|---|---|---|---|---|---|---|---|
| 1 | GPT-6 Astra | xhigh | 42.2% ±1.9 | $1.19 | $28 | 153k | codex_cli 0.154.0 |
| 2 | GPT-6.1 Sol | xhigh | 38.3% ±2.1 | $0.31 | $5.29 | 216k | codex_cli 0.156.0 |
| 3 | Opus 5.5 | xhigh | 31.2% ±1.2 | $0.16 | $10 | 181k | claude_code 2.1.280 |
| 4 | GPT-6.1 Terra | xhigh | 28.1% ±1.5 | $0.14 | $2.86 | 147k | codex_cli 0.164.0 |
| 5 | Fable 5.1 | xhigh | 26.0% ±1.4 | $4.98 | $32 | 251k | claude_code 2.1.270 |
| 6 | Gemini 4 Argon | high | 20.3% ±1.5 | $12 | $16 | 310k | gemini_cli 0.62.0 |
| 7 | Sonnet 5.5 | xhigh | 19.9% ±1.6 | $0.14 | $9.27 | 279k | claude_code 2.1.280 |
| 8 | GPT-6 Sol | xhigh | 19.7% ±1.3 | $0.21 | $6.91 | 144k | codex_cli 0.156.0 |
| 9 | Opus 5 | xhigh | 19.0% ±1.7 | $0.44 | $25 | 288k | claude_code 2.1.270 |
| 10 | Fable 5 | xhigh | 15.6% ±2.0 | $5.95 | $53 | 300k | claude_code 2.1.270 |
| 11 | Muse Wonder (Max) | xhigh | 15.2% ±1.1 | $7.32 | $11 | 169k | codex_cli 0.162.1 |
| 12 | GPT-5.6 Sol | xhigh | 15.0% ±1.4 | $0.17 | $6.28 | 116k | codex_cli 0.154.0 |
| 13 | Haiku 5.5 | xhigh | 11.7% ±1.6 | $0.049 | $2.30 | 315k | claude_code 2.1.280 |
| 14 | DeepSeek V4.1 Flash 1017 | high | 9.2% ±1.2 | free | $0.88 | 830k | codex_cli 0.162.1 |
| 15 | Grok 4.7 | xhigh | 8.5% ±1.2 | $2.50 | $27 | 275k | codex_cli 0.156.0 |
| 16 | GPT-6 Luna | xhigh | 7.9% ±1.0 | $0.024 | $0.32 | 181k | codex_cli 0.156.0 |
| 17 | Kimi K3 | high | 7.5% ±1.4 | $1.81 | $20 | 209k | kimi_code 2.0.0 |
| 18 | Gemini 3.8 Flash | high | 7.4% ±1.0 | $0.16 | $3.94 | 156k | gemini_cli 0.59.0 |
| 19 | Muse Spark 1.3 | xhigh | 6.7% ±1.3 | $2.78 | $5.56 | 280k | codex_cli 0.156.0 |
| 20 | Opus 4.5 | high | 6.5% ±1.5 | $8.70 | $8.70 | 83k | claude_code 2.1.270 |
| 21 | DeepSeek V4.1 Flash | high | 5.9% ±1.8 | $0.053 | $0.74 | 323k | opencode 1.18.30 |
| 22 | GPT-5.6 Luna | xhigh | 5.2% ±1.4 | $0.034 | $0.74 | 99k | codex_cli 0.154.0 |
| 23 | GPT-5.2 | xhigh | 5.0% ±1.0 | $9.29 | $9.29 | 265k | codex_cli 0.156.0 |
| 24 | Mistral Large 4 | high | 4.4% ±1.4 | $1.37 | $29 | 1.1M | opencode 1.18.30 |
| 25 | Inkling | xhigh | 3.9% ±1.0 | $0.12 | $1.33 | 20k | opencode 1.18.30 |