Muse Glimmer

There is no maintained public leaderboard (the project site says it is under construction), so almost all current numbers are self-reported by providers. Results with tools such as a code interpreter are much higher than without: Alibaba reports 83.7% without and 90.2% with tools for Qwen3.8-27B. Top models now exceed the human baseline, so differences at the top are small and may reflect label noise. Descriptive-question scores are a different metric and must not be recorded here.

Elo is a relative rating: values change as models are added and the scale is re-anchored, so always store the date a rating was read. For example, Anthropic's launch page gives 1846 for Claude Opus 5.5, while the leaderboard on 2026-10-12 shows 1866 (Max, Default Fallback). Earlier versions (Google cites "GDPVal-AA v2") are not comparable. Judging is done by an LLM, not by human experts. Anthropic models are listed with "Default Fallback", meaning refused steps are routed to a fallback model. AA-Briefcase and GDP.pdf are separate evaluations.

Results differ a lot by setting: with tools versus without tools (Anthropic reports 45.9% without tools and 57.4% with tools for Claude Haiku 5.5), and the full set versus the text-only subset. Providers use different judge models and harnesses (Scale uses o3-mini; Alibaba states GPT-4o). Derived sets such as HLE-Verified (a third-party revision with 1,811 scored items), HLE-Rolling and HLE-Diamond are different item sets and are not comparable. The questions are public, so contamination is possible, and community reviews have reported flawed items.

Scores measure voter preference, which can be influenced by answer style and length. The leaderboard offers a style-control adjustment, so record which view was used. Scores shift with every update, so a value is only meaningful together with its leaderboard date. Entries marked "Preliminary" or with few votes have wide confidence intervals. The same model can appear several times with different reasoning-effort suffixes (for example -high or -max). Category views (Coding, Math, Creative Writing and others) are different rankings from Overall.