Benchmark · Reasoning
Humanity's Last Exam
accuracy % (%) Higher is better Benchmark homepage
What it measures
Humanity's Last Exam (HLE) is a closed-ended academic benchmark of 2,500 expert-written questions across a broad range of subjects. It is organised by the Center for AI Safety (CAIS) and Scale AI and was published in Nature in 2026. About 14% of the questions require understanding a diagram or figure, and 24% are multiple choice. Scale Labs runs the public leaderboard, which uses o3-mini as the answer judge and also reports calibration error.
Caveats:
Results differ a lot by setting: with tools versus without tools (Anthropic reports 45.9% without tools and 57.4% with tools for Claude Haiku 5.5), and the full set versus the text-only subset. Providers use different judge models and harnesses (Scale uses o3-mini; Alibaba states GPT-4o). Derived sets such as HLE-Verified (a third-party revision with 1,811 scored items), HLE-Rolling and HLE-Diamond are different item sets and are not comparable. The questions are public, so contamination is possible, and community reviews have reported flawed items.
Results
| Model | Score | Setting | Reported by | Measured | Verified | Source |
|---|---|---|---|---|---|---|
| Claude Opus 5.5 Anthropic | 67.7 % | with tools (web search/fetch, programmatic tool calling, code execution); max effort; Claude Opus 4.6 grader | Self-reported | — | 2026-10-12 | Claude Opus 5.5 System Card (Anthropic, 2026-09-22) |
| Claude Fable 5.1 Anthropic | 65.6 % | with tools; max effort; Claude Opus 4.6 grader (Opus 5.5 system card comparison) | Self-reported | — | 2026-10-12 | Claude Opus 5.5 System Card (Anthropic, 2026-09-22) |
| Claude Fable 5.1 Anthropic | 65 % | with tools (restricted web fetch, web search, programmatic tool calling, code execution); max effort; Claude Opus 4.6 grader | Self-reported | — | 2026-10-12 | Claude Fable 5.1 & Claude Mythos 5.1 System Card (Anthropic, 2026-09-01) |
| Claude Sonnet 5.5 Anthropic | 64.5 % | with tools (web search/fetch, programmatic tool calling, code execution); max effort; Claude Opus 4.6 grader | Self-reported | — | 2026-10-12 | Claude Sonnet 5.5 System Card (Anthropic, 2026-09-28) |
| Claude Opus 5.5 Anthropic | 64.4 % | no tools; max effort; full set (2,500 q), Claude Opus 4.6 grader | Self-reported | — | 2026-10-12 | Claude Opus 5.5 System Card (Anthropic, 2026-09-22) |
| DeepSeek-V4.1-Flash DeepSeek | 63.9 % | with tools, Pass@1; max effort | Self-reported | — | 2026-10-25 | deepseek-ai/DeepSeek-V4.1-Flash model card (Hugging Face) |
| Muse Spark 1.1 Meta | 62.1 % | full set (2,500 questions), with tools (bash and browser; blocklist: huggingface.co, kaggle.com, paperswithcode.com, lmsys.org); xhigh effort; o3-mini judge | Self-reported | — | 2026-10-25 | Muse Spark 1.1 Evaluation Report (Meta, 9 Jul 2026) |
| Muse Spark Open Meta | 61 % | question set not stated, with tools (bash and browser; blocklist: huggingface.co, kaggle.com, paperswithcode.com, lmsys.org); xhigh effort; judge not stated | Self-reported | — | 2026-10-25 | meta-models/Muse-Spark-Open-816B-A42B model card |
| Claude Fable 5.1 Anthropic | 60.9 % | no tools; max effort; full set (2,500 q), Claude Opus 4.6 grader | Self-reported | — | 2026-10-12 | Claude Fable 5.1 & Claude Mythos 5.1 System Card (Anthropic, 2026-09-01) |
| DeepSeek-V4-Pro DeepSeek | 60 % | with tools, Pass@1; max effort | Self-reported | — | 2026-10-25 | deepseek-ai/DeepSeek-V4.1-Flash model card (Hugging Face) |
| Claude Haiku 5.5 Anthropic | 57.4 % | with tools (web search/fetch, programmatic tool calling, code execution); max effort; Claude Opus 4.6 grader | Self-reported | — | 2026-10-12 | Claude Haiku 5.5 System Card (Anthropic, 2026-10-07) |
| GPT-6 Astra OpenAI | 57.2 % | with tools; highest score across tested efforts | Self-reported | — | 2026-10-25 | GPT-6 Astra: A new generation of intelligence | OpenAI |
| Claude Sonnet 5.5 Anthropic | 56.9 % | no tools; max effort; full set (2,500 q), Claude Opus 4.6 grader | Self-reported | — | 2026-10-12 | Claude Sonnet 5.5 System Card (Anthropic, 2026-09-28) |
| Qwen3.8-Max Alibaba Cloud (Qwen) | 56.2 % | with tools; effort not stated; judge not stated | Self-reported | — | 2026-10-25 | Qwen3.8-Max: A New Bar for Coding and Cowork (Qwen blog, 2026-08-03) |
| GPT-6 Astra OpenAI | 54.8 % | Scale Labs harness, o3-mini judge | Third party | — | 2026-10-25 | Humanity's Last Exam leaderboard (Scale Labs) |
| GPT-6 Astra OpenAI | 54.17 % | text-only subset; Scale Labs harness, o3-mini judge | Third party | — | 2026-10-25 | Humanity's Last Exam (Text Only) leaderboard (Scale Labs) |
| Muse Spark 1.1 Meta | 52.2 % | full set (2,500 questions), no tools; xhigh effort; o3-mini judge | Self-reported | — | 2026-10-25 | Muse Spark 1.1 Evaluation Report (Meta, 9 Jul 2026) |
| Gemini 3.1 Pro Google | 51.4 % | full set (text + MM), with tools (search with blocklist + code); high effort (Thinking High) | Self-reported | — | 2026-10-25 | Gemini 3.1 Pro model card - Google DeepMind |
| Claude Fable 5.1 Anthropic | 46.5 % | xhigh effort; Scale Labs harness, o3-mini judge (tools not stated) | Third party | — | 2026-10-12 | Humanity's Last Exam leaderboard (Scale Labs) |
| Gemini 3.1 Pro Google | 46.44 % | high effort (thinking high); Scale Labs harness, o3-mini judge; tools not stated | Third party | — | 2026-10-25 | Humanity's Last Exam leaderboard (Scale Labs) |
| Claude Sonnet 5.5 Anthropic | 46.12 % | xhigh effort; Scale Labs harness, o3-mini judge (tools not stated) | Third party | — | 2026-10-12 | Humanity's Last Exam leaderboard (Scale Labs) |
| Claude Haiku 5.5 Anthropic | 45.9 % | no tools; max effort; full set (2,500 q), Claude Opus 4.6 grader | Self-reported | — | 2026-10-12 | Claude Haiku 5.5 System Card (Anthropic, 2026-10-07) |
| Gemini 3.8 Flash Google | 45.4 % | settings not stated (tools, effort, judge and subset not given) | Self-reported | — | 2026-10-25 | Developer's guide to Gemini 3.8 Flash (Google Cloud) |
| Gemini 3.8 Flash Google | 44.52 % | effort not shown; Scale Labs harness, o3-mini judge; tools not stated | Third party | — | 2026-10-25 | Humanity's Last Exam leaderboard (Scale Labs) |
| Gemini 3.1 Pro Google | 44.4 % | full set (text + MM), no tools; high effort (Thinking High) | Self-reported | — | 2026-10-25 | Gemini 3.1 Pro model card - Google DeepMind |
| DeepSeek-V4.1-Flash DeepSeek | 44 % | snapshot DeepSeek-V4.1-Flash-1017; no tools, text-only subset; effort not stated | Self-reported | — | 2026-10-25 | deepseek-ai/DeepSeek-V4.1-Flash-1017 model card (Hugging Face) |
| Qwen3.8-Max Alibaba Cloud (Qwen) | 43.6 % | no tools; effort not stated; judge not stated | Self-reported | — | 2026-10-25 | Qwen3.8-Max: A New Bar for Coding and Cowork (Qwen blog, 2026-08-03) |
| DeepSeek-V4-Pro DeepSeek | 42.7 % | no tools, text-only subset, Pass@1; max effort | Self-reported | — | 2026-10-25 | deepseek-ai/DeepSeek-V4.1-Flash model card (Hugging Face) |
| DeepSeek-V4.1-Flash DeepSeek | 42.6 % | snapshot DeepSeek-V4.1-Flash-1017; no tools, full set; effort not stated | Self-reported | — | 2026-10-25 | deepseek-ai/DeepSeek-V4.1-Flash-1017 model card (Hugging Face) |
| DeepSeek-V4.1-Flash DeepSeek | 39.1 % | no tools, text-only subset, Pass@1; max effort | Self-reported | — | 2026-10-25 | deepseek-ai/DeepSeek-V4.1-Flash model card (Hugging Face) |
| DeepSeek-V4.1-Flash DeepSeek | 36.8 % | no tools, full set, Pass@1; max effort | Self-reported | — | 2026-10-25 | deepseek-ai/DeepSeek-V4.1-Flash model card (Hugging Face) |
| Qwen3.8-Omni-Flash Alibaba Cloud (Qwen) | 36.5 % | tools not stated; effort not stated; GPT-4o judge | Self-reported | — | 2026-10-25 | Qwen3.8-Omni-Flash: Omni Senses. Agentic Delivery. (Qwen blog, September 2026) |
| Qwen3.8-Flash Alibaba Cloud (Qwen) | 35.9 % | tools not stated; effort not stated; GPT-4o judge | Self-reported | — | 2026-10-25 | Qwen3.8-Omni-Flash: Omni Senses. Agentic Delivery. (Qwen blog, September 2026) |
| Qwen3.7-Plus Alibaba Cloud (Qwen) | 34.7 % | tools not stated; effort not stated; GPT-4o judge | Self-reported | — | 2026-10-25 | Qwen/Qwen3.8-27B - Hugging Face model card |
| Qwen3.8-27B Alibaba Cloud (Qwen) | 30.8 % | tools not stated; effort not stated; GPT-4o judge | Self-reported | — | 2026-10-25 | Qwen/Qwen3.8-27B - Hugging Face model card |
| Claude Haiku 5.5 Anthropic | 30.12 % | effort not stated; Scale Labs harness, o3-mini judge (tools not stated) | Third party | — | 2026-10-12 | Humanity's Last Exam leaderboard (Scale Labs) |
| Gemma 4 31B Google | 26.5 % | with search; thinking mode not stated | Self-reported | — | 2026-10-25 | Model card - Gemma Documentation |
| Muse Glimmer Meta | 22 % | text-only subset (2,158 questions), no tools; high reasoning strength; result sourced by Meta from Artificial Analysis | Self-reported | — | 2026-10-25 | meta-models/Muse-Glimmer-30B model card (Meta Superintelligence Lab) |
| Gemma 4 31B Google | 19.5 % | no tools; thinking mode not stated | Self-reported | — | 2026-10-25 | Model card - Gemma Documentation |
| gpt-oss-120b OpenAI | 19 % | with tools; high effort | Self-reported | — | 2026-10-25 | gpt-oss-120b & gpt-oss-20b Model Card (OpenAI, arXiv 2508.10925) |
| Gemini 3.5 Flash-Lite Google | 18 % | settings not stated (tools, effort, judge and subset not given) | Self-reported | — | 2026-10-25 | What's new in Gemini 3.6 Flash and 3.5 Flash-Lite |
| gpt-oss-20b OpenAI | 17.3 % | with tools; high effort | Self-reported | — | 2026-10-25 | gpt-oss-120b & gpt-oss-20b Model Card (OpenAI, arXiv 2508.10925) |
| Gemini 3.1 Flash-Lite Google | 16 % | full set (text + MM), no tools; high effort | Self-reported | — | 2026-10-25 | Gemini 3.1 Flash-Lite model card - Google DeepMind |
| gpt-oss-120b OpenAI | 15.48 % | text-only subset; Scale Labs harness, o3-mini judge | Third party | — | 2026-10-25 | Humanity's Last Exam (Text Only) leaderboard (Scale Labs) |
| gpt-oss-120b OpenAI | 14.9 % | no tools; high effort | Self-reported | — | 2026-10-25 | gpt-oss-120b & gpt-oss-20b Model Card (OpenAI, arXiv 2508.10925) |
| Claude Sonnet 4.5 Anthropic | 13.72 % | thinking; Scale Labs harness, o3-mini judge (tools not stated) | Third party | — | 2026-10-12 | Humanity's Last Exam leaderboard (Scale Labs) |
| gpt-oss-120b OpenAI | 11.3 % | with tools; medium effort | Self-reported | — | 2026-10-25 | gpt-oss-120b & gpt-oss-20b Model Card (OpenAI, arXiv 2508.10925) |
| gpt-oss-20b OpenAI | 10.9 % | no tools; high effort | Self-reported | — | 2026-10-25 | gpt-oss-120b & gpt-oss-20b Model Card (OpenAI, arXiv 2508.10925) |
| gpt-oss-20b OpenAI | 9.73 % | text-only subset; Scale Labs harness, o3-mini judge | Third party | — | 2026-10-25 | Humanity's Last Exam (Text Only) leaderboard (Scale Labs) |
| gpt-oss-120b OpenAI | 9.1 % | with tools; low effort | Self-reported | — | 2026-10-25 | gpt-oss-120b & gpt-oss-20b Model Card (OpenAI, arXiv 2508.10925) |
| gpt-oss-20b OpenAI | 8.8 % | with tools; medium effort | Self-reported | — | 2026-10-25 | gpt-oss-120b & gpt-oss-20b Model Card (OpenAI, arXiv 2508.10925) |
| gpt-oss-120b OpenAI | 8.6 % | no tools; medium effort | Self-reported | — | 2026-10-25 | gpt-oss-120b & gpt-oss-20b Model Card (OpenAI, arXiv 2508.10925) |
| Claude Sonnet 4.5 Anthropic | 7.52 % | no thinking label; Scale Labs harness, o3-mini judge (tools not stated) | Third party | — | 2026-10-12 | Humanity's Last Exam leaderboard (Scale Labs) |
| gpt-oss-20b OpenAI | 7 % | no tools; medium effort | Self-reported | — | 2026-10-25 | gpt-oss-120b & gpt-oss-20b Model Card (OpenAI, arXiv 2508.10925) |
| gpt-oss-20b OpenAI | 6.3 % | with tools; low effort | Self-reported | — | 2026-10-25 | gpt-oss-120b & gpt-oss-20b Model Card (OpenAI, arXiv 2508.10925) |
| Llama 4 Maverick Meta | 5.68 % | full set; tools not stated; Scale Labs harness, o3-mini judge | Third party | — | 2026-10-25 | Humanity's Last Exam leaderboard (Scale Labs) |
| gpt-oss-120b OpenAI | 5.2 % | no tools; low effort | Self-reported | — | 2026-10-25 | gpt-oss-120b & gpt-oss-20b Model Card (OpenAI, arXiv 2508.10925) |
| Amazon Nova Pro Amazon | 4.4 % | full set (2,500 questions); Scale Labs harness, o3-mini judge; tools not stated | Third party | — | 2026-10-12 | Humanity's Last Exam leaderboard (Scale Labs) |
| gpt-oss-20b OpenAI | 4.2 % | no tools; low effort | Self-reported | — | 2026-10-25 | gpt-oss-120b & gpt-oss-20b Model Card (OpenAI, arXiv 2508.10925) |
| Amazon Nova Lite Amazon | 3.64 % | full set (2,500 questions); Scale Labs harness, o3-mini judge; tools not stated | Third party | — | 2026-10-12 | Humanity's Last Exam leaderboard (Scale Labs) |
Sorted by score, without rank numbers: each result was measured under its own setting (effort level, harness, tools, test version), as stated by the source. Scores are only directly comparable between rows with the same setting. Self-reported results come from the provider and have not been reproduced independently.
Sources
- Humanity's Last Exam leaderboard (Scale Labs)
- Humanity's Last Exam
- Claude Opus 5.5 System Card (Anthropic, 2026-09-22)
- Claude Fable 5.1 & Claude Mythos 5.1 System Card (Anthropic, 2026-09-01)
- Claude Sonnet 5.5 System Card (Anthropic, 2026-09-28)
- deepseek-ai/DeepSeek-V4.1-Flash model card (Hugging Face)
- Muse Spark 1.1 Evaluation Report (Meta, 9 Jul 2026)
- meta-models/Muse-Spark-Open-816B-A42B model card
- Claude Haiku 5.5 System Card (Anthropic, 2026-10-07)
- GPT-6 Astra: A new generation of intelligence | OpenAI
- Qwen3.8-Max: A New Bar for Coding and Cowork (Qwen blog, 2026-08-03)
- Humanity's Last Exam (Text Only) leaderboard (Scale Labs)
- Gemini 3.1 Pro model card - Google DeepMind
- Developer's guide to Gemini 3.8 Flash (Google Cloud)
- deepseek-ai/DeepSeek-V4.1-Flash-1017 model card (Hugging Face)
- Qwen3.8-Omni-Flash: Omni Senses. Agentic Delivery. (Qwen blog, September 2026)
- Qwen/Qwen3.8-27B - Hugging Face model card
- Model card - Gemma Documentation
- meta-models/Muse-Glimmer-30B model card (Meta Superintelligence Lab)
- gpt-oss-120b & gpt-oss-20b Model Card (OpenAI, arXiv 2508.10925)
- What's new in Gemini 3.6 Flash and 3.5 Flash-Lite
- Gemini 3.1 Flash-Lite model card - Google DeepMind