Muse Spark Open

The official leaderboard uses mini-swe-agent for all models and shows whole-number percentages with margins of about ±1 to ±6 points. Providers often report scores from their own scaffolds or effort settings; DeepSeek reports 65.5% to 74.2% for the same model depending on the scaffold. DeepSWE v1 scores are not comparable with v1.1. With only 113 tasks, small differences are within noise.

FrontierMath was developed with funding from OpenAI, which has exclusive access to a subset of the problems. Scores from v1 and from Tier 4 (a separate 43-problem expansion set) are not comparable with Tiers 1-3 (v2). The best models score about 94%, so the benchmark separates mid-size models better than it separates frontier models. Epoch's numbers come from its own harness (Python tool, 1M-token limit), which can differ from providers' internal setups. Margins are about ±1.4 to ±2.6 points.

Elo is a relative rating: values change as models are added and the scale is re-anchored, so always store the date a rating was read. For example, Anthropic's launch page gives 1846 for Claude Opus 5.5, while the leaderboard on 2026-10-12 shows 1866 (Max, Default Fallback). Earlier versions (Google cites "GDPVal-AA v2") are not comparable. Judging is done by an LLM, not by human experts. Anthropic models are listed with "Default Fallback", meaning refused steps are routed to a fallback model. AA-Briefcase and GDP.pdf are separate evaluations.

Results differ a lot by setting: with tools versus without tools (Anthropic reports 45.9% without tools and 57.4% with tools for Claude Haiku 5.5), and the full set versus the text-only subset. Providers use different judge models and harnesses (Scale uses o3-mini; Alibaba states GPT-4o). Derived sets such as HLE-Verified (a third-party revision with 1,811 scored items), HLE-Rolling and HLE-Diamond are different item sets and are not comparable. The questions are public, so contamination is possible, and community reviews have reported flawed items.