Benchmark · Multimodal
CharXiv Reasoning
accuracy % (%) Higher is better Benchmark homepage
What it measures
CharXiv tests how well multimodal models understand realistic charts, using 2,323 charts taken from arXiv papers and verified by human experts. Its reasoning questions require combining information across complex visual elements of a chart; the separate descriptive questions cover basic chart elements. It was built by researchers at Princeton Language and Intelligence with co-authors from the University of Wisconsin-Madison and the University of Hong Kong (NeurIPS 2024). The human baseline is 80.5%.
There is no maintained public leaderboard (the project site says it is under construction), so almost all current numbers are self-reported by providers. Results with tools such as a code interpreter are much higher than without: Alibaba reports 83.7% without and 90.2% with tools for Qwen3.8-27B. Top models now exceed the human baseline, so differences at the top are small and may reflect label noise. Descriptive-question scores are a different metric and must not be recorded here.
Results
| Model | Score | Setting | Reported by | Measured | Verified | Source |
|---|---|---|---|---|---|---|
| Qwen3.8-Max Alibaba Cloud (Qwen) | 93.5 % | reasoning questions (RQ), with code interpreter (CI); effort not stated | Self-reported | — | 2026-10-25 | Qwen3.8-Max: A New Bar for Coding and Cowork (Qwen blog, 2026-08-03) |
| Qwen3.8-Omni-Flash Alibaba Cloud (Qwen) | 91.4 % | reasoning questions (RQ), with code interpreter (CI); effort not stated | Self-reported | — | 2026-10-25 | Qwen3.8-Omni-Flash: Omni Senses. Agentic Delivery. (Qwen blog, September 2026) |
| Qwen3.8-Flash Alibaba Cloud (Qwen) | 90.6 % | reasoning questions (RQ), with code interpreter (CI); effort not stated | Self-reported | — | 2026-10-25 | Qwen3.8-Omni-Flash: Omni Senses. Agentic Delivery. (Qwen blog, September 2026) |
| Qwen3.8-27B Alibaba Cloud (Qwen) | 90.2 % | reasoning questions (RQ), with code interpreter (CI); effort not stated | Self-reported | — | 2026-10-25 | Qwen/Qwen3.8-27B - Hugging Face model card |
| Gemini 3.6 Flash Google | 89.4 % | reasoning questions; with tools (search and code execution); effort not stated | Self-reported | — | 2026-10-25 | Gemini 3.6 Flash model card - Google DeepMind |
| Muse Spark 1.1 Meta | 88.4 % | reasoning questions, validation set (1,000 questions), with tools (server-side code execution, no search); xhigh effort; gpt-oss-120b judge | Self-reported | — | 2026-10-25 | Muse Spark 1.1 Evaluation Report (Meta, 9 Jul 2026) |
| Qwen3.8-Max Alibaba Cloud (Qwen) | 88.4 % | reasoning questions (RQ), without code interpreter (CI); effort not stated | Self-reported | — | 2026-10-25 | Qwen3.8-Max: A New Bar for Coding and Cowork (Qwen blog, 2026-08-03) |
| Gemini 3.8 Flash Google | 86.2 % | reasoning questions; no tools; effort not stated | Self-reported | — | 2026-10-25 | Gemini 3.8 Flash model card - Google DeepMind |
| Qwen3.7-Plus Alibaba Cloud (Qwen) | 85.9 % | reasoning questions (RQ), with code interpreter (CI); effort not stated | Self-reported | — | 2026-10-25 | Qwen3.8-Max: A New Bar for Coding and Cowork (Qwen blog, 2026-08-03) |
| Qwen3.7-Plus Alibaba Cloud (Qwen) | 85.8 % | reasoning questions (RQ), without code interpreter (CI); effort not stated | Self-reported | — | 2026-10-25 | Qwen3.8-Max: A New Bar for Coding and Cowork (Qwen blog, 2026-08-03) |
| Gemini 3.6 Flash Google | 85.2 % | reasoning questions; no tools; effort not stated | Self-reported | — | 2026-10-25 | Gemini 3.6 Flash model card - Google DeepMind |
| Qwen3.8-Flash Alibaba Cloud (Qwen) | 84.6 % | reasoning questions (RQ), without code interpreter (CI); effort not stated | Self-reported | — | 2026-10-25 | Qwen3.8-Omni-Flash: Omni Senses. Agentic Delivery. (Qwen blog, September 2026) |
| Qwen3.8-27B Alibaba Cloud (Qwen) | 83.7 % | reasoning questions (RQ), without code interpreter (CI); effort not stated | Self-reported | — | 2026-10-25 | Qwen/Qwen3.8-27B - Hugging Face model card |
| Qwen3.8-Omni-Flash Alibaba Cloud (Qwen) | 83.5 % | reasoning questions (RQ), without code interpreter (CI); effort not stated | Self-reported | — | 2026-10-25 | Qwen3.8-Omni-Flash: Omni Senses. Agentic Delivery. (Qwen blog, September 2026) |
| Gemini 3.1 Pro Google | 83.3 % | reasoning questions; no tools; effort not stated | Self-reported | — | 2026-10-25 | Gemini 3.6 Flash model card - Google DeepMind |
| Gemini 3.1 Pro Google | 83.2 % | reasoning questions; with tools (search and code execution); effort not stated | Self-reported | — | 2026-10-25 | Gemini 3.6 Flash model card - Google DeepMind |
| Muse Glimmer Meta | 78.8 % | reasoning questions, validation set (1,000 questions), tools not stated; high reasoning strength; gpt-oss-120b judge, mean of 4 attempts | Self-reported | — | 2026-10-25 | meta-models/Muse-Glimmer-30B model card (Meta Superintelligence Lab) |
| Gemini 3.5 Flash-Lite Google | 76.5 % | reasoning questions; with tools (search and code execution); high effort | Self-reported | — | 2026-10-25 | Gemini 3.5 Flash-Lite model card - Google DeepMind |
| Gemini 3.1 Flash-Lite Google | 75.6 % | reasoning questions; with tools (search and code execution); effort not stated | Self-reported | — | 2026-10-25 | Gemini 3.5 Flash-Lite model card - Google DeepMind |
| Gemini 3.5 Flash-Lite Google | 74.5 % | reasoning questions; no tools; high effort | Self-reported | — | 2026-10-25 | Gemini 3.5 Flash-Lite model card - Google DeepMind |
| Gemini 3.1 Flash-Lite Google | 73.2 % | reasoning questions; no tools; high effort | Self-reported | — | 2026-10-25 | Gemini 3.1 Flash-Lite model card - Google DeepMind |
| Command A+ Cohere | 52.7 % | CharXiv reasoning questions; tools, split and thinking mode not stated ('standard methodology') | Self-reported | — | 2026-10-25 | Introducing Command A+ | Cohere |
| Command A Vision Cohere | 46.9 % | CharXiv reasoning questions; tools, split and thinking mode not stated ('standard methodology') | Self-reported | — | 2026-10-25 | Introducing Command A+ | Cohere |
Sorted by score, without rank numbers: each result was measured under its own setting (effort level, harness, tools, test version), as stated by the source. Scores are only directly comparable between rows with the same setting. Self-reported results come from the provider and have not been reproduced independently.
Sources
- CharXiv (arXiv 2406.18521)
- CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs
- Qwen3.8-Max: A New Bar for Coding and Cowork (Qwen blog, 2026-08-03)
- Qwen3.8-Omni-Flash: Omni Senses. Agentic Delivery. (Qwen blog, September 2026)
- Qwen/Qwen3.8-27B - Hugging Face model card
- Gemini 3.6 Flash model card - Google DeepMind
- Muse Spark 1.1 Evaluation Report (Meta, 9 Jul 2026)
- Gemini 3.8 Flash model card - Google DeepMind
- meta-models/Muse-Glimmer-30B model card (Meta Superintelligence Lab)
- Gemini 3.5 Flash-Lite model card - Google DeepMind
- Gemini 3.1 Flash-Lite model card - Google DeepMind
- Introducing Command A+ | Cohere