Benchmarks · coding
WeirdML v2
17 ML tasks where models write Python pipelines from scratch (shape inference, chess winners, Kolmogorov shuffles); mean accuracy across tasks.
111 models · 127 results · data as of 2026-11-06 · htihle.github.io ↗
Leaderboard
GPT-6 Astra Pro xhigh
Heavy line: best score at each price. Thin lines join one model's effort levels. Cost uses each model's median subscription plan, or API price with no plan data. 87 more models under All models in settings.
All results
127 rows| # | Model | Effort | WeirdML v2 | Subscription $ / task | API $ / task |
|---|---|---|---|---|---|
| 1 | GPT-6 Astra Pro | xhigh | 93.6% | $29 | $29 |
| 2 | GPT-6 Astra | xhigh | 93.3% | $0.33 | $7.75 |
| 3 | Fable 5.1 | xhigh | 92.9% | $1.82 | $12 |
| 4 | GPT-6 Astra | high | 92.9% | $0.13 | $2.98 |
| 5 | Fable 5.1 | high | 92.3% | $0.50 | $3.20 |
| 6 | Fable 5 | xhigh | 91.9% | $0.95 | $8.42 |
| 7 | Opus 5 | xhigh | 91.8% | $0.082 | $4.74 |
| 8 | Opus 5 | high | 91.6% | $0.038 | $2.19 |
| 9 | GPT-5.6 Sol Pro | xhigh | 89.4% | $3.87 | $3.87 |
| 10 | GPT-5.6 Sol | max | 88.8% | $0.035 | $1.25 |
| 11 | Fable 5 | max | 87.8% | $0.35 | $3.11 |
| 12 | GPT-5.6 Sol | xhigh | 87.0% | $0.064 | $2.31 |
| 13 | Opus 5 | max | 86.3% | $0.025 | $1.42 |
| 14 | GPT-5.5 | xhigh | 84.9% | $0.071 | $2.57 |
| 15 | Gemini 3.8 Flash | max | 84.8% | $0.021 | $0.51 |
| 16 | Opus 4.8 | xhigh | 82.9% | $0.041 | $2.35 |
| 17 | Kimi K3 | max | 82.6% | $0.19 | $2.05 |
| 18 | GPT-5.6 Terra | max | 78.3% | $0.007 | $0.25 |
| 19 | Opus 4.6 | max | 77.9% | $5.78 | $5.78 |
| 20 | GPT-5.3 Codex | xhigh | 77.9% | $2.45 | $2.45 |
| 21 | GPT-5.4 | xhigh | 77.7% | $5.72 | $5.72 |
| 22 | Opus 4.7 | max | 76.4% | $1.82 | $1.82 |
| 23 | Opus 4.8 | medium | 76.0% | $0.015 | $0.85 |
| 24 | Opus 4.7 | xhigh | 75.4% | $7.95 | $7.95 |
| 25 | GLM 5.3 | max | 75.4% | $0.056 | $0.54 |
| 26 | Qwen 3.8 | xhigh | 75.2% | $1.42 | $1.42 |
| 27 | GPT-5.2 | xhigh | 72.2% | $2.05 | $2.05 |
| 28 | Gemini 3.1 Pro | high | 72.1% | $1.36 | $1.36 |
| 29 | Opus 4.8 | max | 70.5% | $0.008 | $0.48 |
| 30 | GLM 5.2 | xhigh | 70.1% | $0.015 | $0.51 |