Muse Spark 1.2

Leaderboard versions (for example 1.0.5 and 1.0.6) are not comparable. Some results use fallback routing for refused steps: for "Claude Fable 5.1 (with Opus 5 Fallback)", Opus 5 handled 260 of 657 tasks. Artificial Analysis runs a separate leaderboard, AutomationBench-AA, which uses a guardrail-adjusted metric; its scores are not comparable with Zapier's. The partial_credit metric is for diagnostics only. The leaderboard shows no per-row evaluation dates.

The official leaderboard uses mini-swe-agent for all models and shows whole-number percentages with margins of about ±1 to ±6 points. Providers often report scores from their own scaffolds or effort settings; DeepSeek reports 65.5% to 74.2% for the same model depending on the scaffold. DeepSWE v1 scores are not comparable with v1.1. With only 113 tasks, small differences are within noise.

Elo is a relative rating: values change as models are added and the scale is re-anchored, so always store the date a rating was read. For example, Anthropic's launch page gives 1846 for Claude Opus 5.5, while the leaderboard on 2026-10-12 shows 1866 (Max, Default Fallback). Earlier versions (Google cites "GDPVal-AA v2") are not comparable. Judging is done by an LLM, not by human experts. Anthropic models are listed with "Default Fallback", meaning refused steps are routed to a fallback model. AA-Briefcase and GDP.pdf are separate evaluations.

Numbers are only comparable when release, task set (full or offline), metric (binary or partial), step budget (150, 300 or 500) and agent approach (standard or batch tool calls) all match. Providers usually quote partial scores, which are much higher than binary completion. Claude Opus 5 at max effort on v2.1 scores 44.33% binary and 77.67% partial. The verified leaderboard is run by the maintainers and covers few current models, so most numbers are self-reported. The original OSWorld and OSWorld-Verified (369 tasks) are a different benchmark and are not recorded here.

Scores measure voter preference, which can be influenced by answer style and length. The leaderboard offers a style-control adjustment, so record which view was used. Scores shift with every update, so a value is only meaningful together with its leaderboard date. Entries marked "Preliminary" or with few votes have wide confidence intervals. The same model can appear several times with different reasoning-effort suffixes (for example -high or -max). Category views (Coding, Math, Creative Writing and others) are different rankings from Overall.