Benchmarks · agentic
AutomationBench (Zapier)
Zapier's agent benchmark (v1.0.6): end-to-end business workflows across 47 real tools in six functions, graded strictly on the final environment state (every assertion must pass). Private held-out set.
40 models · 118 results · data as of 2026-11-06 · zapier.com ↗
Leaderboard
Gemini 4 Carbon high
Heavy line: best score at each price. Thin lines join one model's effort levels. Cost uses each model's median subscription plan, or API price with no plan data. 27 more models under All models in settings.
All results
118 rows| # | Model | Effort | AutomationBench (Zapier) | Subscription $ / task | API $ / task |
|---|---|---|---|---|---|
| 1 | Gemini 4 Carbon | high | 53.9% | $0.76 | $2.12 |
| 2 | Gemini 4 Carbon | medium | 53.0% | $0.59 | $1.65 |
| 3 | Gemini 4 Argon | high | 51.3% | $1.21 | $1.70 |
| 4 | Gemini 4 Argon | medium | 50.1% | $1.10 | $1.54 |
| 5 | Sonnet 5.5 | max | 44.8% | $0.017 | $1.14 |
| 6 | Opus 5.5 | max | 42.5% | $0.022 | $1.44 |
| 7 | GPT-6 Astra | max | 41.4% | $0.074 | $1.73 |
| 8 | GPT-6 Astra | xhigh | 39.0% | $0.064 | $1.50 |
| 9 | GPT-6 Astra | high | 37.1% | $0.062 | $1.44 |
| 10 | Sonnet 5.5 | xhigh | 36.8% | $0.007 | $0.48 |
| 11 | Opus 5.5 | xhigh | 35.8% | $0.013 | $0.89 |
| 12 | GPT-6 Astra | medium | 34.1% | $0.054 | $1.27 |
| 13 | GPT-6 Sol | xhigh | 33.2% | $0.008 | $0.27 |
| 14 | Opus 5.5 | high | 33.0% | $0.011 | $0.71 |
| 15 | GPT-6 Sol | max | 32.0% | $0.011 | $0.34 |
| 16 | Gemini 4 Carbon | low | 31.5% | $0.36 | $1.02 |
| 17 | GPT-6 Sol | high | 31.2% | $0.007 | $0.24 |
| 18 | Gemini 3.7 Flash | high | 30.4% | $0.079 | $0.61 |
| 19 | GPT-6 Astra | low | 30.3% | $0.046 | $1.08 |
| 20 | Gemini 3.8 Flash | high | 29.7% | $0.025 | $0.62 |
| 21 | Gemini 3.8 Flash | medium | 29.7% | $0.022 | $0.55 |
| 22 | Opus 5.5 | medium | 29.5% | $0.010 | $0.65 |
| 23 | Sonnet 5.5 | high | 29.4% | $0.005 | $0.36 |
| 24 | GPT-5.6 Sol | max | 28.8% | $0.018 | $0.67 |
| 25 | Opus 5 | max | 26.9% | $0.053 | $3.05 |
| 26 | GPT-6 Sol | medium | 26.9% | $0.006 | $0.21 |
| 27 | GPT-5.6 Sol | xhigh | 26.3% | $0.020 | $0.74 |
| 28 | Opus 5 | xhigh | 25.3% | $0.020 | $1.15 |
| 29 | GPT-6 Astra | none | 25.3% | $0.055 | $1.28 |
| 30 | GPT-5.6 Sol | high | 24.8% | $0.018 | $0.64 |