Benchmarks · reasoning
CritPt v2 Hard Subset
The private 40-question hard subset of CritPt v2: the 70 challenges minus the 30 that seven frontier models solved in all five attempts at launch. Code-execution tool in an offline sandbox; accuracy.
Category
45 models · 93 results · data as of 2026-11-06 · artificialanalysis.ai ↗
Leaderboard
GPT-6 Astra medium
Heavy line: best score at each price. Thin lines join one model's effort levels. Cost uses each model's median subscription plan, or API price with no plan data. 27 more models under All models in settings.
More from this benchmark
4 published measuresScore vs any published measure4 measures: sub-scores, splits, counts
27 more models under All models in settings.
Breakdown
darker is better within each column · click a column to sort| Model | ||||
|---|---|---|---|---|
| GPT-6 Astra medium | 908,872 | 123,055 | 127,589 | 667,438 |
| GPT-6.1 Sol max | 2,077,656 | 156,223 | 597,534 | 1,480,931 |
| Gemini 4 Carbon high | 73,733,125 | 1,175,506 | 4,060,423 | 67,661,911 |
| Opus 5.5 max | 30,507,258 | 398,764 | 3,487,513 | 26,587,314 |
| GPT-6.1 Terra max | 10,774,559 | 203,652 | 1,331,553 | 9,331,077 |
| GPT-6 Sol max | 8,677,859 | 155,309 | 955,400 | 7,634,549 |
| Fable 5.1 high | 8,586,844 | 248,297 | 1,245,272 | 7,100,786 |
| GPT-5.6 Sol max | 10,515,506 | 198,032 | 1,031,028 | 9,350,938 |
| Sonnet 5.5 max | 59,677,774 | 494,212 | 5,143,006 | 54,052,155 |
| Gemini 4 Argon high | 41,985,516 | 609,862 | 2,250,466 | 38,702,215 |
| Muse Wonder (Max) | 38,235,687 | 374,528 | 2,280,907 | 35,346,049 |
| GPT-5.6 Terra max | 14,926,552 | 209,625 | 1,322,671 | 13,462,769 |
| Opus 5 max | 30,304,120 | 516,064 | 2,721,626 | 26,899,477 |
| Kimi K3.1 max | 21,827,525 | 261,498 | 1,651,699 | 19,794,627 |
| Muse Spark 1.3 max | 45,639,342 | 638,644 | 2,929,271 | 41,983,634 |
| Haiku 5.5 xhigh | 25,088,885 | 319,679 | 3,124,863 | 21,799,143 |
| Gemini 3.8 Flash high | 68,635,173 | 776,247 | 1,883,528 | 65,687,870 |
| Grok 4.7 low | 29,470,202 | 421,977 | 1,387,075 | 27,422,031 |
| GPT-6 Luna max | 16,104,278 | 179,065 | 1,772,017 | 14,793,738 |
| Kimi K3 max | 23,567,341 | 267,772 | 1,495,257 | 21,667,352 |
| Grok 4.6 xhigh | 31,688,109 | 431,868 | 1,832,634 | 29,463,732 |
| Hy4 | 26,952,435 | 293,740 | 2,848,905 | 24,018,959 |
| Inkling 2 Small | 93,421,185 | 1,206,531 | 4,223,072 | 87,941,331 |
| MiMo V2.6 Pro | 67,497,157 | 450,009 | 4,201,659 | 62,535,477 |
| GLM 5.3 max | 98,920,363 | 553,517 | 5,825,702 | 92,412,450 |
All results
93 rows| # | Model | Effort | CritPt v2 Hard Subset | Subscription $ / task | API $ / task | Harness |
|---|---|---|---|---|---|---|
| 1 | GPT-6 Astra | medium | 72.0% | $0.017 | $0.41 | Stirrup |
| 2 | GPT-6 Astra | high | 71.5% | $0.031 | $0.71 | Stirrup |
| 3 | GPT-6 Astra | xhigh | 71.0% | $0.036 | $0.85 | Stirrup |
| 4 | GPT-6 Astra | max | 70.5% | $0.049 | $1.14 | Stirrup |
| 5 | GPT-6.1 Sol | max | 69.5% | $0.013 | $0.23 | Stirrup |
| 6 | GPT-6.1 Sol | xhigh | 69.5% | $0.010 | $0.17 | Stirrup |
| 7 | Gemini 4 Carbon | high | 69.0% | $1.39 | $3.91 | Stirrup |
| 8 | Gemini 4 Carbon | medium | 69.0% | $1.10 | $3.09 | Stirrup |
| 9 | GPT-6.1 Sol | high | 68.5% | $0.008 | $0.13 | Stirrup |
| 10 | Opus 5.5 | max | 68.0% | $0.039 | $2.58 | Stirrup |
| 11 | GPT-6.1 Terra | max | 68.0% | $0.013 | $0.25 | Stirrup |
| 12 | Opus 5.5 | xhigh | 67.5% | $0.015 | $0.97 | Stirrup |
| 13 | GPT-6.1 Sol | medium | 67.0% | $0.004 | $0.071 | Stirrup |
| 14 | GPT-6 Sol | max | 66.5% | $0.012 | $0.38 | Stirrup |
| 15 | Gemini 4 Carbon | low | 65.0% | $1.09 | $3.06 | Stirrup |
| 16 | GPT-6 Astra | low | 63.0% | $0.010 | $0.24 | Stirrup |
| 17 | GPT-6.1 Sol | low | 62.5% | $0.003 | $0.043 | Stirrup |
| 18 | Fable 5.1 | high | 62.0% | $0.37 | $2.39 | Stirrup |
| 19 | GPT-6.1 Terra | high | 61.5% | $0.003 | $0.069 | Stirrup |
| 20 | GPT-6.1 Terra | xhigh | 60.5% | $0.006 | $0.12 | Stirrup |
| 21 | Fable 5.1 | xhigh | 60.0% | $0.68 | $4.33 | Stirrup |
| 22 | GPT-5.6 Sol | max | 60.0% | Stirrup | ||
| 23 | Opus 5.5 | high | 59.5% | $0.006 | $0.38 | Stirrup |
| 24 | Sonnet 5.5 | max | 59.0% | $0.029 | $1.93 | Stirrup |
| 25 | Fable 5.1 | medium | 58.5% | $0.22 | $1.38 | Stirrup |
| 26 | Sonnet 5.5 | xhigh | 57.5% | $0.007 | $0.45 | Stirrup |
| 27 | GPT-6.1 Terra | medium | 56.5% | $0.002 | $0.040 | Stirrup |
| 28 | Opus 5.5 | medium | 54.0% | $0.004 | $0.25 | Stirrup |
| 29 | Fable 5.1 | max | 52.5% | $0.71 | $4.55 | Stirrup |
| 30 | Gemini 4 Argon | high | 52.5% | $1.67 | $2.35 | Stirrup |