Benchmark · Other
AutomationBench
% tasks completed correctly (%) Higher is better Benchmark homepage
What it measures
AutomationBench, built by Zapier, tests AI agents on end-to-end business workflows across Sales, Marketing, Operations, Support, Finance and HR, using 47 simulated apps with about 500 API endpoints. Agents receive one request, must find the relevant records and business rules themselves, and cannot ask questions. Scoring is deterministic and checks only the final state of the environment, with no LLM judge. The headline metric, task_completed_correctly, is strict pass/fail on a held-out private task set; the leaderboard currently shows version 1.0.6.
Caveats:
Leaderboard versions (for example 1.0.5 and 1.0.6) are not comparable. Some results use fallback routing for refused steps: for "Claude Fable 5.1 (with Opus 5 Fallback)", Opus 5 handled 260 of 657 tasks. Artificial Analysis runs a separate leaderboard, AutomationBench-AA, which uses a guardrail-adjusted metric; its scores are not comparable with Zapier's. The partial_credit metric is for diagnostics only. The leaderboard shows no per-row evaluation dates.
Results
| Model | Score | Setting | Reported by | Measured | Verified | Source |
|---|---|---|---|---|---|---|
| DeepSeek-V4.1-Flash DeepSeek | 60.5 % | snapshot DeepSeek-V4.1-Flash-1017; v1.0.6, task set not stated (probably the public evaluation set); effort not stated | Self-reported | — | 2026-10-25 | deepseek-ai/DeepSeek-V4.1-Flash-1017 model card (Hugging Face) |
| DeepSeek-V4.1-Flash DeepSeek | 54.8 % | public evaluation set, v1.0.6, Pass@1; max effort; official scaffold | Self-reported | — | 2026-10-25 | DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression (DeepSeek-AI, arXiv 2609.19969, submitted 2026-09-17) |
| Gemini 4 Argon Google | 51.29 % | v1.0.6; high effort | Third party | — | 2026-10-25 | AutomationBench AI benchmark leaderboard (Zapier) |
| Gemini 4 Argon Google | 50.08 % | v1.0.6; medium effort | Third party | — | 2026-10-25 | AutomationBench AI benchmark leaderboard (Zapier) |
| Muse Spark 1.3 Meta | 49.6 % | public v3 task set (600 workflows), pass@1; max effort; benchmark's automation tools | Self-reported | — | 2026-10-25 | Muse Spark 1.3 model page |
| Muse Spark 1.3 Meta | 48.3 % | public v3 task set (600 workflows), pass@1; xhigh effort; benchmark's automation tools | Self-reported | — | 2026-10-25 | Muse Spark 1.3 Evaluation Methodology (PDF) |
| Claude Sonnet 5.5 Anthropic | 44.75 % | v1.0.6; default fallbacks; max effort | Third party | — | 2026-10-12 | AutomationBench AI benchmark leaderboard (Zapier) |
| Claude Sonnet 5.5 Anthropic | 44.7 % | v1.0.6; default fallbacks; max effort; run by Zapier | Self-reported | — | 2026-10-12 | Claude Sonnet 5.5 System Card (Anthropic, 2026-09-28) |
| DeepSeek-V4-Pro DeepSeek | 43.2 % | public evaluation set, v1.0.6, Pass@1; max effort; official scaffold | Self-reported | — | 2026-10-25 | DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression (DeepSeek-AI, arXiv 2609.19969, submitted 2026-09-17) |
| Claude Opus 5.5 Anthropic | 42.5 % | v1.0.6; default fallbacks (refused tasks re-run); max effort; run by Zapier | Self-reported | — | 2026-10-12 | Claude Sonnet 5.5 System Card (Anthropic, 2026-09-28) |
| Claude Opus 5.5 Anthropic | 42.47 % | v1.0.6; default fallbacks; max effort | Third party | — | 2026-10-12 | AutomationBench AI benchmark leaderboard (Zapier) |
| GPT-6 Astra OpenAI | 41.4 % | max effort | Self-reported | — | 2026-10-25 | GPT-6 Astra: A new generation of intelligence | OpenAI |
| GPT-6 Astra OpenAI | 41.4 % | v1.0.6; max effort | Third party | — | 2026-10-25 | AutomationBench AI benchmark leaderboard (Zapier) |
| Claude Opus 5.5 Anthropic | 40 % | no fallback models (safeguard interventions counted as failures); max effort; run by Zapier | Self-reported | — | 2026-10-12 | Claude Opus 5.5 announcement |
| GPT-6 Astra OpenAI | 39 % | xhigh effort | Self-reported | — | 2026-10-25 | GPT-6 Astra: A new generation of intelligence | OpenAI |
| GPT-6 Astra OpenAI | 38.96 % | v1.0.6; xhigh effort | Third party | — | 2026-10-25 | AutomationBench AI benchmark leaderboard (Zapier) |
| Muse Spark 1.2 Meta | 38.2 % | public v3 task set (600 workflows), pass@1; xhigh effort; benchmark's automation tools | Self-reported | — | 2026-10-25 | Muse Spark 1.3 model page |
| GPT-6 Astra OpenAI | 37.14 % | v1.0.6; high effort | Third party | — | 2026-10-25 | AutomationBench AI benchmark leaderboard (Zapier) |
| GPT-6 Astra OpenAI | 37.1 % | high effort | Self-reported | — | 2026-10-25 | GPT-6 Astra: A new generation of intelligence | OpenAI |
| Claude Sonnet 5.5 Anthropic | 36.83 % | v1.0.6; default fallbacks; xhigh effort | Third party | — | 2026-10-12 | AutomationBench AI benchmark leaderboard (Zapier) |
| GPT-6.1 Sol OpenAI | 36.1 % | v1.0.6; max effort | Self-reported | — | 2026-10-25 | Introducing GPT-6.1 Sol | OpenAI |
| Claude Opus 5.5 Anthropic | 35.77 % | v1.0.6; default fallbacks; xhigh effort | Third party | — | 2026-10-12 | AutomationBench AI benchmark leaderboard (Zapier) |
| GPT-6.1 Sol OpenAI | 35.5 % | v1.0.6; xhigh effort | Self-reported | — | 2026-10-25 | Introducing GPT-6.1 Sol | OpenAI |
| GPT-6 Astra OpenAI | 34.1 % | medium effort | Self-reported | — | 2026-10-25 | GPT-6 Astra: A new generation of intelligence | OpenAI |
| GPT-6 Astra OpenAI | 34.09 % | v1.0.6; medium effort | Third party | — | 2026-10-25 | AutomationBench AI benchmark leaderboard (Zapier) |
| GPT-6 Sol OpenAI | 33.2 % | v1.0.6; xhigh effort | Self-reported | — | 2026-10-25 | Introducing GPT-6 Sol and Luna | OpenAI |
| GPT-6 Sol OpenAI | 33.2 % | v1.0.6; xhigh effort | Third party | — | 2026-10-25 | AutomationBench AI benchmark leaderboard (Zapier) |
| GPT-6.1 Sol OpenAI | 33.2 % | v1.0.6; high effort | Self-reported | — | 2026-10-25 | Introducing GPT-6.1 Sol | OpenAI |
| Claude Opus 5.5 Anthropic | 33.03 % | v1.0.6; default fallbacks; high effort | Third party | — | 2026-10-12 | AutomationBench AI benchmark leaderboard (Zapier) |
| GPT-6 Sol OpenAI | 32 % | v1.0.6; max effort | Self-reported | — | 2026-10-25 | Introducing GPT-6 Sol and Luna | OpenAI |
| GPT-6 Sol OpenAI | 32 % | v1.0.6; max effort | Third party | — | 2026-10-25 | AutomationBench AI benchmark leaderboard (Zapier) |
| DeepSeek-V4-Pro DeepSeek | 31.8 % | AutomationBench (Public), version not stated; effort not stated | Self-reported | — | 2026-10-25 | deepseek-ai/DeepSeek-V4-Pro-0813 model card (Hugging Face) |
| GPT-6.1 Sol OpenAI | 31.7 % | v1.0.6; medium effort | Self-reported | — | 2026-10-25 | Introducing GPT-6.1 Sol | OpenAI |
| Claude Fable 5.1 Anthropic | 31.4 % | max effort; Zapier leaderboard (production safeguards on) | Self-reported | — | 2026-10-12 | Claude Fable 5.1 & Claude Mythos 5.1 System Card (Anthropic, 2026-09-01) |
| Claude Fable 5.1 Anthropic | 31.4 % | v1.0.6; with Opus 5 fallback | Third party | — | 2026-10-12 | AutomationBench AI benchmark leaderboard (Zapier) |
| GPT-6 Sol OpenAI | 31.2 % | v1.0.6; high effort | Self-reported | — | 2026-10-25 | Introducing GPT-6 Sol and Luna | OpenAI |
| GPT-6 Sol OpenAI | 31.2 % | v1.0.6; high effort | Third party | — | 2026-10-25 | AutomationBench AI benchmark leaderboard (Zapier) |
| GPT-6 Astra OpenAI | 30.3 % | low effort | Self-reported | — | 2026-10-25 | GPT-6 Astra: A new generation of intelligence | OpenAI |
| GPT-6 Astra OpenAI | 30.29 % | v1.0.6; low effort | Third party | — | 2026-10-25 | AutomationBench AI benchmark leaderboard (Zapier) |
| Gemini 3.8 Flash Google | 29.68 % | v1.0.6; medium effort | Third party | — | 2026-10-25 | AutomationBench AI benchmark leaderboard (Zapier) |
| Gemini 3.8 Flash Google | 29.68 % | v1.0.6; high effort | Third party | — | 2026-10-25 | AutomationBench AI benchmark leaderboard (Zapier) |
| Claude Opus 5.5 Anthropic | 29.53 % | v1.0.6; default fallbacks; medium effort | Third party | — | 2026-10-12 | AutomationBench AI benchmark leaderboard (Zapier) |
| Claude Sonnet 5.5 Anthropic | 29.38 % | v1.0.6; default fallbacks; high effort | Third party | — | 2026-10-12 | AutomationBench AI benchmark leaderboard (Zapier) |
| Qwen3.8-Max Alibaba Cloud (Qwen) | 27.3 % | 600-task public subset, Pass@1; effort not stated; harness not stated | Self-reported | — | 2026-10-25 | Qwen3.8-Max: A New Bar for Coding and Cowork (Qwen blog, 2026-08-03) |
| GPT-6 Sol OpenAI | 26.9 % | v1.0.6; medium effort | Self-reported | — | 2026-10-25 | Introducing GPT-6 Sol and Luna | OpenAI |
| GPT-6 Sol OpenAI | 26.9 % | v1.0.6; medium effort | Third party | — | 2026-10-25 | AutomationBench AI benchmark leaderboard (Zapier) |
| GPT-6 Astra OpenAI | 25.27 % | v1.0.6; none effort | Third party | — | 2026-10-25 | AutomationBench AI benchmark leaderboard (Zapier) |
| GPT-6.1 Sol OpenAI | 24.7 % | v1.0.6; low effort | Self-reported | — | 2026-10-25 | Introducing GPT-6.1 Sol | OpenAI |
| Claude Opus 5.5 Anthropic | 24.2 % | v1.0.6; default fallbacks; low effort | Third party | — | 2026-10-12 | AutomationBench AI benchmark leaderboard (Zapier) |
| Claude Sonnet 5.5 Anthropic | 23.29 % | v1.0.6; default fallbacks; medium effort | Third party | — | 2026-10-12 | AutomationBench AI benchmark leaderboard (Zapier) |
| Claude Fable 5.1 Anthropic | 22.37 % | v1.0.6; max effort | Third party | — | 2026-10-12 | AutomationBench AI benchmark leaderboard (Zapier) |
| Claude Fable 5.1 Anthropic | 21.31 % | v1.0.6; xhigh effort | Third party | — | 2026-10-12 | AutomationBench AI benchmark leaderboard (Zapier) |
| GPT-6 Sol OpenAI | 21.2 % | v1.0.6; low effort | Self-reported | — | 2026-10-25 | Introducing GPT-6 Sol and Luna | OpenAI |
| GPT-6 Sol OpenAI | 21.2 % | v1.0.6; low effort | Third party | — | 2026-10-25 | AutomationBench AI benchmark leaderboard (Zapier) |
| GPT-6 Luna OpenAI | 20.7 % | v1.0.6; max effort | Self-reported | — | 2026-10-25 | Introducing GPT-6 Sol and Luna | OpenAI |
| GPT-6 Luna OpenAI | 20.7 % | v1.0.6; max effort | Third party | — | 2026-10-25 | AutomationBench AI benchmark leaderboard (Zapier) |
| Muse Spark 1.3 Meta | 20.7 % | v1.0.6 (official held-out private task set); max effort | Third party | — | 2026-10-25 | AutomationBench AI benchmark leaderboard (Zapier) |
| Claude Sonnet 5.5 Anthropic | 18.57 % | v1.0.6; default fallbacks; between_tools thinking | Third party | — | 2026-10-12 | AutomationBench AI benchmark leaderboard (Zapier) |
| Claude Sonnet 5.5 Anthropic | 18.26 % | v1.0.6; default fallbacks; low effort | Third party | — | 2026-10-12 | AutomationBench AI benchmark leaderboard (Zapier) |
| Gemini 3.6 Flash Google | 17.05 % | v1.0.6; high effort | Third party | — | 2026-10-25 | AutomationBench AI benchmark leaderboard (Zapier) |
| Gemini 3.6 Flash Google | 16.44 % | v1.0.6; medium effort | Third party | — | 2026-10-25 | AutomationBench AI benchmark leaderboard (Zapier) |
| Gemini 3.6 Flash Google | 15.68 % | v1.0.6; low effort | Third party | — | 2026-10-25 | AutomationBench AI benchmark leaderboard (Zapier) |
| GPT-6 Luna OpenAI | 14.5 % | v1.0.6; high effort | Self-reported | — | 2026-10-25 | Introducing GPT-6 Sol and Luna | OpenAI |
| GPT-6 Luna OpenAI | 14.46 % | v1.0.6; high effort | Third party | — | 2026-10-25 | AutomationBench AI benchmark leaderboard (Zapier) |
| GPT-6 Luna OpenAI | 12.63 % | v1.0.6; xhigh effort | Third party | — | 2026-10-25 | AutomationBench AI benchmark leaderboard (Zapier) |
| GPT-6 Luna OpenAI | 12.6 % | v1.0.6; xhigh effort | Self-reported | — | 2026-10-25 | Introducing GPT-6 Sol and Luna | OpenAI |
| Claude Haiku 5.5 Anthropic | 12.02 % | v1.0.6; default fallbacks; max effort | Third party | — | 2026-10-12 | AutomationBench AI benchmark leaderboard (Zapier) |
| Claude Haiku 5.5 Anthropic | 10.5 % | v1.0.6; default fallbacks; high effort | Third party | — | 2026-10-12 | AutomationBench AI benchmark leaderboard (Zapier) |
| Claude Haiku 5.5 Anthropic | 9.89 % | v1.0.6; default fallbacks; xhigh effort | Third party | — | 2026-10-12 | AutomationBench AI benchmark leaderboard (Zapier) |
| GPT-6 Luna OpenAI | 9.44 % | v1.0.6; medium effort | Third party | — | 2026-10-25 | AutomationBench AI benchmark leaderboard (Zapier) |
| GPT-6 Luna OpenAI | 9.4 % | v1.0.6; medium effort | Self-reported | — | 2026-10-25 | Introducing GPT-6 Sol and Luna | OpenAI |
| GPT-6 Sol OpenAI | 9.1 % | v1.0.6; none effort | Third party | — | 2026-10-25 | AutomationBench AI benchmark leaderboard (Zapier) |
| Gemini 3.1 Pro Google | 8.68 % | v1.0.6; high effort | Third party | — | 2026-10-25 | AutomationBench AI benchmark leaderboard (Zapier) |
| Claude Haiku 5.5 Anthropic | 8.52 % | v1.0.6; default fallbacks; medium effort | Third party | — | 2026-10-12 | AutomationBench AI benchmark leaderboard (Zapier) |
| Gemini 3.1 Pro Google | 7 % | v1.0.6; medium effort | Third party | — | 2026-10-25 | AutomationBench AI benchmark leaderboard (Zapier) |
| Claude Haiku 5.5 Anthropic | 6.09 % | v1.0.6; default fallbacks; low effort | Third party | — | 2026-10-12 | AutomationBench AI benchmark leaderboard (Zapier) |
| Gemini 3.1 Pro Google | 4.57 % | v1.0.6; low effort | Third party | — | 2026-10-25 | AutomationBench AI benchmark leaderboard (Zapier) |
| Gemini 3.5 Flash-Lite Google | 3.96 % | v1.0.6; medium effort | Third party | — | 2026-10-25 | AutomationBench AI benchmark leaderboard (Zapier) |
| Qwen3.7-Plus Alibaba Cloud (Qwen) | 3.7 % | v1.0.6; effort not shown | Third party | — | 2026-10-25 | AutomationBench AI benchmark leaderboard (Zapier) |
| Gemini 3.5 Flash-Lite Google | 3.35 % | v1.0.6; high effort | Third party | — | 2026-10-25 | AutomationBench AI benchmark leaderboard (Zapier) |
| Gemma 4 31B Google | 1.7 % | v1.0.6; max effort | Third party | — | 2026-10-25 | AutomationBench AI benchmark leaderboard (Zapier) |
| GPT-6 Luna OpenAI | 1.22 % | v1.0.6; low effort | Third party | — | 2026-10-25 | AutomationBench AI benchmark leaderboard (Zapier) |
| GPT-6 Luna OpenAI | 1.2 % | v1.0.6; low effort | Self-reported | — | 2026-10-25 | Introducing GPT-6 Sol and Luna | OpenAI |
| GPT-6 Luna OpenAI | 0.3 % | v1.0.6; none effort | Third party | — | 2026-10-25 | AutomationBench AI benchmark leaderboard (Zapier) |
| gpt-oss-120b OpenAI | 0.15 % | v1.0.6; high effort | Third party | — | 2026-10-25 | AutomationBench AI benchmark leaderboard (Zapier) |
| Gemini 3.5 Flash-Lite Google | 0 % | v1.0.6; low effort | Third party | — | 2026-10-25 | AutomationBench AI benchmark leaderboard (Zapier) |
Sorted by score, without rank numbers: each result was measured under its own setting (effort level, harness, tools, test version), as stated by the source. Scores are only directly comparable between rows with the same setting. Self-reported results come from the provider and have not been reproduced independently.
Sources
- AutomationBench (arXiv 2604.18934)
- AutomationBench AI benchmark leaderboard (Zapier)
- deepseek-ai/DeepSeek-V4.1-Flash-1017 model card (Hugging Face)
- DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression (DeepSeek-AI, arXiv 2609.19969, submitted 2026-09-17)
- Muse Spark 1.3 model page
- Muse Spark 1.3 Evaluation Methodology (PDF)
- Claude Sonnet 5.5 System Card (Anthropic, 2026-09-28)
- GPT-6 Astra: A new generation of intelligence | OpenAI
- Claude Opus 5.5 announcement
- Introducing GPT-6.1 Sol | OpenAI
- Introducing GPT-6 Sol and Luna | OpenAI
- deepseek-ai/DeepSeek-V4-Pro-0813 model card (Hugging Face)
- Claude Fable 5.1 & Claude Mythos 5.1 System Card (Anthropic, 2026-09-01)
- Qwen3.8-Max: A New Bar for Coding and Cowork (Qwen blog, 2026-08-03)