Benchmark · Coding
DeepSWE v1.1
% resolved (pass@1) (%) Higher is better Benchmark homepage
What it measures
DeepSWE measures coding agents on original, long-horizon software engineering tasks: 113 tasks across 91 repositories and five programming languages. The tasks were written from scratch to limit training-data contamination, and their verifiers test software behaviour rather than implementation details. Version 1.1 kept the tasks but changed execution and grading: the agent's committed patch is applied in a fresh container and tested there. DeepSWE is maintained by Datacurve, whose leaderboard runs every model on the mini-swe-agent harness.
Caveats:
The official leaderboard uses mini-swe-agent for all models and shows whole-number percentages with margins of about ±1 to ±6 points. Providers often report scores from their own scaffolds or effort settings; DeepSeek reports 65.5% to 74.2% for the same model depending on the scaffold. DeepSWE v1 scores are not comparable with v1.1. With only 113 tasks, small differences are within noise.
Results
| Model | Score | Setting | Reported by | Measured | Verified | Source |
|---|---|---|---|---|---|---|
| Muse Wonder Full Meta | 79 % | v1.1 (113 tasks), pass@1; max effort; result Meta attributes to Datacurve, harness not stated (not on the public Datacurve board as of 2026-10-25) | Self-reported | — | 2026-10-25 | Muse Wonder Model Card (PDF) |
| Muse Wonder Meta | 78.8 % | v1.1 (113 tasks), pass@1; max effort; result Meta attributes to Datacurve, harness not stated (not on the public Datacurve board as of 2026-10-25) | Self-reported | — | 2026-10-25 | Muse Wonder Model Card (PDF) |
| Gemini 4 Argon Google | 77.9 % | v1.1; high effort; mini-swe-agent harness | Self-reported | — | 2026-10-25 | Gemini 4 Argon model card - Google DeepMind |
| DeepSeek-V4.1-Flash DeepSeek | 77 % | snapshot DeepSeek-V4.1-Flash-1017; v1.1; effort not stated; Mini-SWE-V2 harness | Self-reported | — | 2026-10-25 | deepseek-ai/DeepSeek-V4.1-Flash-1017 model card (Hugging Face) |
| Muse Spark 1.3 Meta | 75.4 % | v1.1 (113 tasks), task pass rate; max effort; mini-swe-agent (run by Meta) | Self-reported | — | 2026-10-25 | Muse Spark 1.3 model page |
| GPT-6.1 Sol OpenAI | 75.2 % | v1.1; high effort | Self-reported | — | 2026-10-25 | Introducing GPT-6.1 Sol | OpenAI |
| Claude Opus 5.5 Anthropic | 74.2 % | v1.1; max effort; scaffold not stated; mean of 5 trials | Self-reported | — | 2026-10-12 | Claude Opus 5.5 System Card (Anthropic, 2026-09-22) |
| DeepSeek-V4.1-Flash DeepSeek | 74.2 % | Resolved, N=8 samples per task; max effort; mini-SWE harness (mini-swe-agent) | Self-reported | — | 2026-10-25 | deepseek-ai/DeepSeek-V4.1-Flash model card (Hugging Face) |
| GPT-6 Astra OpenAI | 74.1 % | v1.1; xhigh effort | Self-reported | — | 2026-10-25 | GPT-6 Astra: A new generation of intelligence | OpenAI |
| Gemini 3.8 Flash Google | 74 % | v1.1; high effort; mini-swe-agent harness | Third party | — | 2026-10-25 | DeepSWE leaderboard (Datacurve) |
| GPT-6 Astra OpenAI | 74 % | v1.1; xhigh effort; mini-swe-agent | Third party | — | 2026-10-25 | DeepSWE leaderboard (Datacurve) |
| Gemini 3.8 Flash Google | 73.7 % | v1.1; high effort; mini-swe-agent harness | Self-reported | — | 2026-10-25 | Gemini 3.8 Flash model card - Google DeepMind |
| GPT-6 Astra OpenAI | 73.2 % | v1.1; high effort | Self-reported | — | 2026-10-25 | GPT-6 Astra: A new generation of intelligence | OpenAI |
| GPT-6 Astra OpenAI | 73 % | v1.1; max effort | Self-reported | — | 2026-10-25 | GPT-6 Astra: A new generation of intelligence | OpenAI |
| GPT-6 Astra OpenAI | 73 % | v1.1; high effort; mini-swe-agent | Third party | — | 2026-10-25 | DeepSWE leaderboard (Datacurve) |
| GPT-6 Astra OpenAI | 73 % | v1.1; max effort; mini-swe-agent | Third party | — | 2026-10-25 | DeepSWE leaderboard (Datacurve) |
| GPT-6 Astra OpenAI | 73 % | v1.1; medium effort; mini-swe-agent | Third party | — | 2026-10-25 | DeepSWE leaderboard (Datacurve) |
| GPT-6.1 Sol OpenAI | 73 % | v1.1; medium effort | Self-reported | — | 2026-10-25 | Introducing GPT-6.1 Sol | OpenAI |
| GPT-6 Astra OpenAI | 72.8 % | v1.1; medium effort | Self-reported | — | 2026-10-25 | GPT-6 Astra: A new generation of intelligence | OpenAI |
| DeepSeek-V4.1-Flash DeepSeek | 72.6 % | Resolved, N=8 samples per task; max effort; DeepSeek Harness (Minimal mode) | Self-reported | — | 2026-10-25 | DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression (DeepSeek-AI, arXiv 2609.19969, submitted 2026-09-17) |
| GPT-6.1 Sol OpenAI | 71.9 % | v1.1; xhigh effort | Self-reported | — | 2026-10-25 | Introducing GPT-6.1 Sol | OpenAI |
| GPT-6.1 Sol OpenAI | 71.9 % | v1.1; max effort | Self-reported | — | 2026-10-25 | Introducing GPT-6.1 Sol | OpenAI |
| Claude Sonnet 5.5 Anthropic | 71 % | v1.1; max effort; scaffold not stated; mean of 5 trials | Self-reported | — | 2026-10-12 | Claude Sonnet 5.5 System Card (Anthropic, 2026-09-28) |
| Gemini 3.8 Flash Google | 71 % | v1.1; medium effort; mini-swe-agent harness | Third party | — | 2026-10-25 | DeepSWE leaderboard (Datacurve) |
| Grok 4.7 xAI | 71 % | v1.1; high effort; mini-SWE-agent harness | Self-reported | — | 2026-10-25 | Model Card: Grok 4.7 (Revision 2026-09-21) |
| DeepSeek-V4.1-Flash DeepSeek | 70.5 % | Resolved, N=8 samples per task; max effort; DeepSeek Harness (Standard mode) | Self-reported | — | 2026-10-25 | DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression (DeepSeek-AI, arXiv 2609.19969, submitted 2026-09-17) |
| DeepSeek-V4.1-Flash DeepSeek | 69.8 % | Resolved, N=8 samples per task; max effort; Claude Code v2.1.251 | Self-reported | — | 2026-10-25 | DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression (DeepSeek-AI, arXiv 2609.19969, submitted 2026-09-17) |
| GPT-6 Sol OpenAI | 68.8 % | v1.1; max effort | Self-reported | — | 2026-10-25 | Introducing GPT-6 Sol and Luna | OpenAI |
| DeepSeek-V4.1-Flash DeepSeek | 67.6 % | Resolved, N=8 samples per task; max effort; DeepSeek Harness (PTC mode) | Self-reported | — | 2026-10-25 | DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression (DeepSeek-AI, arXiv 2609.19969, submitted 2026-09-17) |
| Claude Fable 5.1 Anthropic | 67.4 % | v1.1; max effort; scaffold not stated; mean of 5 trials | Self-reported | — | 2026-10-12 | Claude Fable 5.1 & Claude Mythos 5.1 System Card (Anthropic, 2026-09-01) |
| GPT-6 Astra OpenAI | 67 % | v1.1; low effort | Self-reported | — | 2026-10-25 | GPT-6 Astra: A new generation of intelligence | OpenAI |
| GPT-6 Astra OpenAI | 67 % | v1.1; low effort; mini-swe-agent | Third party | — | 2026-10-25 | DeepSWE leaderboard (Datacurve) |
| Grok 4.6 xAI | 67 % | v1.1; medium effort; mini-swe-agent | Third party | — | 2026-10-25 | DeepSWE leaderboard (Datacurve) |
| Grok 4.6 xAI | 67 % | v1.1; xhigh effort; mini-swe-agent | Third party | — | 2026-10-25 | DeepSWE leaderboard (Datacurve) |
| Grok 4.6 xAI | 67 % | v1.1; xhigh effort; mini-swe-agent harness | Self-reported | — | 2026-10-25 | Model Card: Grok 4.6 (Revision 2026-08-17) |
| GPT-6 Luna OpenAI | 66.6 % | v1.1; max effort | Self-reported | — | 2026-10-25 | Introducing GPT-6 Sol and Luna | OpenAI |
| GPT-6 Sol OpenAI | 66.6 % | v1.1; xhigh effort | Self-reported | — | 2026-10-25 | Introducing GPT-6 Sol and Luna | OpenAI |
| DeepSeek-V4.1-Flash DeepSeek | 66.2 % | Resolved, N=8 samples per task; max effort; Pi v0.84.2 | Self-reported | — | 2026-10-25 | DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression (DeepSeek-AI, arXiv 2609.19969, submitted 2026-09-17) |
| DeepSeek-V4.1-Flash DeepSeek | 66 % | Resolved; reasoning effort 25 (of 100); harness not stated | Self-reported | — | 2026-10-25 | DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression (DeepSeek-AI, arXiv 2609.19969, submitted 2026-09-17) |
| Grok 4.6 xAI | 65.9 % | v1.1; high effort; mini-swe-agent harness | Self-reported | — | 2026-10-25 | Model Card: Grok 4.6 (Revision 2026-08-17) |
| DeepSeek-V4.1-Flash DeepSeek | 65.6 % | Resolved, N=8 samples per task; max effort; Codex v0.147.0 | Self-reported | — | 2026-10-25 | DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression (DeepSeek-AI, arXiv 2609.19969, submitted 2026-09-17) |
| DeepSeek-V4.1-Flash DeepSeek | 65.5 % | Resolved, N=8 samples per task; max effort; OpenCode v1.18.15 | Self-reported | — | 2026-10-25 | DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression (DeepSeek-AI, arXiv 2609.19969, submitted 2026-09-17) |
| GPT-6 Sol OpenAI | 65.3 % | v1.1; high effort | Self-reported | — | 2026-10-25 | Introducing GPT-6 Sol and Luna | OpenAI |
| Grok 4.6 xAI | 65 % | v1.1; high effort; mini-swe-agent | Third party | — | 2026-10-25 | DeepSWE leaderboard (Datacurve) |
| GPT-6.1 Sol OpenAI | 64.4 % | v1.1; low effort | Self-reported | — | 2026-10-25 | Introducing GPT-6.1 Sol | OpenAI |
| DeepSeek-V4-Pro DeepSeek | 62.7 % | Resolved; max effort; DeepSeek Harness (Minimal mode) | Self-reported | — | 2026-10-25 | deepseek-ai/DeepSeek-V4-Pro-0813 model card (Hugging Face) |
| Mistral Large 4 Mistral AI | 61.7 % | v1.1, 'Mistral Large 4 Preview'; effort not stated; Artificial Analysis harness ('AA-DeepSWE 1.1', run privately by AA), not mini-swe-agent | Self-reported | — | 2026-10-25 | Introducing Mistral Large 4 | Mistral |
| GPT-6 Luna OpenAI | 61.3 % | v1.1; xhigh effort | Self-reported | — | 2026-10-25 | Introducing GPT-6 Sol and Luna | OpenAI |
| Muse Spark Open Meta | 60 % | v1.1 (113 tasks); xhigh effort; Muse Code harness; result sourced by Meta from Artificial Analysis | Self-reported | — | 2026-10-25 | meta-models/Muse-Spark-Open-816B-A42B model card |
| GPT-6 Luna OpenAI | 59.3 % | v1.1; high effort | Self-reported | — | 2026-10-25 | Introducing GPT-6 Sol and Luna | OpenAI |
| Muse Spark 1.2 Meta | 59.3 % | v1.1 (113 tasks), pass@1 averaged over 5 attempts; xhigh effort; Muse Code harness (Meta internal framework, Daytona sandboxes) | Self-reported | — | 2026-10-25 | Introducing Muse Code and Muse Spark 1.2 |
| Qwen3.8-Flash Alibaba Cloud (Qwen) | 58.7 % | v1.1; effort not stated; best of Claude Code and mini-SWE-agent harnesses, temp 1.0, top_p 0.95, 256K context | Self-reported | — | 2026-10-25 | Qwen3.8-Omni-Flash: Omni Senses. Agentic Delivery. (Qwen blog, September 2026) |
| Qwen3.8-Omni-Flash Alibaba Cloud (Qwen) | 57.8 % | v1.1; effort not stated; best of Claude Code and mini-SWE-agent harnesses, temp 1.0, top_p 0.95, 256K context | Self-reported | — | 2026-10-25 | Qwen3.8-Omni-Flash: Omni Senses. Agentic Delivery. (Qwen blog, September 2026) |
| Qwen3.8-Max Alibaba Cloud (Qwen) | 57 % | v1.1 (113 tasks); xhigh effort; mini-swe-agent | Third party | — | 2026-10-25 | DeepSWE leaderboard (Datacurve) |
| GPT-6 Sol OpenAI | 56.6 % | v1.1; medium effort | Self-reported | — | 2026-10-25 | Introducing GPT-6 Sol and Luna | OpenAI |
| Qwen3.8-Max Alibaba Cloud (Qwen) | 56.6 % | v1.1; effort not stated; best of Claude Code and mini-SWE-agent harnesses (Claude Code best), temp 1.0, top_p 0.95, 256K context | Self-reported | — | 2026-10-25 | Qwen3.8-Max: A New Bar for Coding and Cowork (Qwen blog, 2026-08-03) |
| Muse Spark 1.2 Meta | 55 % | v1.1 (113 tasks), pass@1; xhigh effort; mini-swe-agent (official leaderboard harness) | Third party | — | 2026-10-25 | DeepSWE leaderboard (Datacurve) |
| Grok 4.5 xAI | 54 % | v1.1; high effort; mini-swe-agent | Third party | — | 2026-10-25 | DeepSWE leaderboard (Datacurve) |
| Muse Spark 1.1 Meta | 53.3 % | v1.1 (113 tasks), pass@1 averaged over 5 attempts; xhigh effort; Meta internal fork of the bash-only mini-swe-agent harness, internet disabled | Self-reported | — | 2026-10-25 | Muse Spark 1.1 Evaluation Report (Meta, 9 Jul 2026) |
| Grok 4.5 xAI | 53 % | v1.1; high effort; mini-swe-agent harness run by Datacurve | Self-reported | — | 2026-10-25 | Model Card: Grok 4.5 (Revision 2026-07-20) |
| Muse Spark 1.1 Meta | 53 % | v1.1 (113 tasks), pass@1; xhigh effort; mini-swe-agent (official leaderboard harness) | Third party | — | 2026-10-25 | DeepSWE leaderboard (Datacurve) |
| Gemini 3.6 Flash Google | 47 % | v1.1; high effort; mini-swe-agent harness | Third party | — | 2026-10-25 | DeepSWE leaderboard (Datacurve) |
| GPT-6 Luna OpenAI | 44.5 % | v1.1; medium effort | Self-reported | — | 2026-10-25 | Introducing GPT-6 Sol and Luna | OpenAI |
| Qwen3.8-27B Alibaba Cloud (Qwen) | 42.2 % | v1.1; effort not stated; Claude Code harness, temp 1.0, top_p 0.95, 256K context | Self-reported | — | 2026-10-25 | Qwen/Qwen3.8-27B - Hugging Face model card |
| Grok 4.6 xAI | 42 % | v1.1; low effort; mini-swe-agent | Third party | — | 2026-10-25 | DeepSWE leaderboard (Datacurve) |
| GPT-6 Sol OpenAI | 37.2 % | v1.1; low effort | Self-reported | — | 2026-10-25 | Introducing GPT-6 Sol and Luna | OpenAI |
| Qwen3.7-Plus Alibaba Cloud (Qwen) | 16.5 % | v1.1; effort not stated; best of Claude Code and mini-SWE-agent harnesses, temp 1.0, top_p 0.95, 256K context | Self-reported | — | 2026-10-25 | Qwen/Qwen3.8-Flash-Next - Hugging Face model card |
| Qwen3.7-Plus Alibaba Cloud (Qwen) | 14.2 % | v1.1; effort not stated; Claude Code harness, temp 1.0, top_p 0.95, 256K context | Self-reported | — | 2026-10-25 | Qwen/Qwen3.8-27B - Hugging Face model card |
| Gemini 3.1 Pro Google | 12 % | v1.1; high effort; mini-swe-agent harness | Third party | — | 2026-10-25 | DeepSWE leaderboard (Datacurve) |
| GPT-6 Luna OpenAI | 2.4 % | v1.1; low effort | Self-reported | — | 2026-10-25 | Introducing GPT-6 Sol and Luna | OpenAI |
Sorted by score, without rank numbers: each result was measured under its own setting (effort level, harness, tools, test version), as stated by the source. Scores are only directly comparable between rows with the same setting. Self-reported results come from the provider and have not been reproduced independently.
Sources
- DeepSWE v1.1
- DeepSWE leaderboard (Datacurve)
- Muse Wonder Model Card (PDF)
- Gemini 4 Argon model card - Google DeepMind
- deepseek-ai/DeepSeek-V4.1-Flash-1017 model card (Hugging Face)
- Muse Spark 1.3 model page
- Introducing GPT-6.1 Sol | OpenAI
- Claude Opus 5.5 System Card (Anthropic, 2026-09-22)
- deepseek-ai/DeepSeek-V4.1-Flash model card (Hugging Face)
- GPT-6 Astra: A new generation of intelligence | OpenAI
- Gemini 3.8 Flash model card - Google DeepMind
- DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression (DeepSeek-AI, arXiv 2609.19969, submitted 2026-09-17)
- Claude Sonnet 5.5 System Card (Anthropic, 2026-09-28)
- Model Card: Grok 4.7 (Revision 2026-09-21)
- Introducing GPT-6 Sol and Luna | OpenAI
- Claude Fable 5.1 & Claude Mythos 5.1 System Card (Anthropic, 2026-09-01)
- Model Card: Grok 4.6 (Revision 2026-08-17)
- deepseek-ai/DeepSeek-V4-Pro-0813 model card (Hugging Face)
- Introducing Mistral Large 4 | Mistral
- meta-models/Muse-Spark-Open-816B-A42B model card
- Introducing Muse Code and Muse Spark 1.2
- Qwen3.8-Omni-Flash: Omni Senses. Agentic Delivery. (Qwen blog, September 2026)
- Qwen3.8-Max: A New Bar for Coding and Cowork (Qwen blog, 2026-08-03)
- Muse Spark 1.1 Evaluation Report (Meta, 9 Jul 2026)
- Model Card: Grok 4.5 (Revision 2026-07-20)
- Qwen/Qwen3.8-27B - Hugging Face model card
- Qwen/Qwen3.8-Flash-Next - Hugging Face model card