How to compare AI models fairly: benchmarks, prices and capabilities

A comparison between AI models only means something if both sides are measured the same way. Benchmark scores come from different testers and settings, list prices hide differences in caching, prompt-length tiers, tokenizers and reasoning, and "1M context" is not always the same number. This guide shows how to line models up fairly and how to read the compare page and our rankings.

In short:

• Read every benchmark score together with who reported it and the exact setting: tools, harness, reasoning effort and benchmark version. A gap of two or three points is often noise.
• Normalize prices for your own input-to-output ratio, your cache hit rate, the provider's tokenizer and the reasoning tokens each model writes. The useful unit is cost per task, not price per token.
• Compare capabilities on equal terms: "unknown" is not "no", and limits, reasoning modes and release status count as much as features.
• Use side-by-side tables and rankings to narrow the field, then decide with a test on your own inputs.

Fix the comparison first

Before looking at numbers, fix the task, the inputs and outputs, the volume and the settings you would use in production. Rule out models that fail a hard requirement, such as a missing input type or a license you can't accept (How to choose an AI model covers these filters). Only the remaining shortlist is worth comparing in detail.

Read benchmark scores with their fine print

A benchmark score is a measurement under specific conditions. Four questions tell you whether two scores are comparable.

Who ran the test?

Many headline numbers are self-reported: the provider ran the test and chose which results to publish. Launch pages also mix sources. Anthropic's page for Claude Opus 5.5 lists 66.4% on Terminal-Bench 4.0 for Opus 5.5 at "xhigh" effort, next to 57.9% for GPT-6 Astra at high effort, a figure taken from OpenAI's own report; the page describes both as each model's highest score ^[1]. Such a table is useful, but it is not a comparison under identical conditions.

Third-party results from a benchmark maintainer or another independent evaluator compare more easily, because one party runs every model the same way. A good sign is a provider that checks its setup against a public leaderboard: Anthropic reports that the leaderboard lists Claude Opus 5 at 51.8% (five trials per task, Claude Code harness) and that its own setup reproduces 52.3% ^[1]. On Best AI Ever, results are labeled "Self-reported" or "Third party", and rankings prefer third-party results where both exist (methodology).

What was the setting?

The setting is part of the score. Three examples from official pages, as of October 2026:

• Tools. Anthropic reports 45.9% for Claude Haiku 5.5 on Humanity's Last Exam without tools and 57.4% with tools ^[2]. Same model, same benchmark, 11.5 points apart.
• Harness. DeepSeek ran the first release of DeepSeek-V4.1-Flash (September 2026) on DeepSWE v1.1 in eight agent scaffolds, all at maximum reasoning effort. Scores range from 65.5% to 74.2%; the headline 74.2% comes from the harness that matches the benchmark's official setup ^[3].
• Version. Terminal-Bench 4.0 recalibrated task resources, removed eight tasks and fixed 19. The maintainers call these breaking changes that require re-running trials ^[4], so a 3.0 score and a 4.0 score don't belong in the same comparison.

Reasoning effort is a fourth variable and often goes unstated. Our compare table shows a benchmark row only when two or more of the selected models have results in the same setting.

Is the gap bigger than the noise?

Anthropic gives a standard error of ±2.6 points for its Opus 5.5 Terminal-Bench result ^[1]. A gap of two or three points is within that range and proves little. OpenAI's evaluation guide adds that models sometimes produce different output from the same input ^[5], so a single run per model is weak evidence either way.

Does the benchmark resemble your task?

Benchmarks measure narrow tasks, and public test questions can leak into training data. OpenAI's evaluation guide separates industry benchmarks, which compare models in isolation, from tests of your own application, and warns against eval datasets that don't reflect your production traffic ^[5]. Use benchmarks to narrow the shortlist, then test the rest on your own examples; Anthropic recommends many automatically graded cases over a few graded by hand ^[6]. If a model grades the answers, watch for bias: OpenAI notes that judge models can favor answers by position and by length, and recommends pairwise or pass/fail grading and controlling for response length ^[5].

Normalize prices before comparing them

Four adjustments turn per-token list prices into a fair comparison.

Use your own input-to-output ratio

A single "blended" price assumes a mix of input and output. Our rankings blend at 3:1 unless a page states otherwise (methodology), but your workload may differ. Take Claude Haiku 5.5 at $0.10 input and $0.50 output per 1M tokens (prompts up to 100,000 tokens) and Ministral 3 14B at $0.20 for both, as of October 2026. The blended price per 1M tokens:

At 3:1 the two cost the same; for input-heavy work Haiku is cheaper, for output-heavy work Ministral is. Measure your real ratio in test runs.

Look past the headline rate

Claude Fable 5.1 and GPT-6 Astra share the same list price: $10 input and $50 output per 1M tokens. Cache hits cost $0.25 on Fable 5.1 and $1.00 on Astra, and cache writes cost $12.50 on both (as of October 2026). For an agent that sends 1 billion input tokens a month, 90% of them cache hits, and receives 50 million output tokens:

• Claude Fable 5.1: $1,000 fresh input + $225 cached input + $2,500 output = $3,725.
• GPT-6 Astra: $1,000 + $900 + $2,500 = $4,400, about 18% more.

That assumes prompts under 272,000 tokens. Above that, Astra bills $20 input and $75 output per 1M tokens, while Fable 5.1 keeps its standard rate across its full 1M window. Such tiers start at different lengths per model, so check each model page. AI token pricing explained covers caching, batch and tier pricing in detail.

Count tokens, not words

A token is not a fixed amount of text. Google puts a Gemini token at about four characters, or 60 to 80 English words per 100 tokens ^[7]. Anthropic notes that the count depends on the tokenizer, the language and the type of content, and that Claude 4.7 and later models produce about 30% more tokens for the same text than earlier models ^[8].

Claude Sonnet 5.5 lists $2 input and $10 output, a third below the $3 and $15 of Claude Sonnet 4.5, which retires on the Claude API on November 30, 2026. With roughly 30% more tokens for the same text, the saving per document is closer to 13% than 33%. Across providers, count a sample of your own inputs with each provider's token counter, or compare billed usage from test runs.

Include the reasoning tokens

Reasoning models write thinking tokens before the visible answer, and you pay for them. OpenAI bills reasoning tokens as output tokens and doesn't show them through the API ^[9]. Anthropic bills the full thinking tokens even when the response shows only a summary ^[10], and Google labels its Gemini 3.1 Pro output price as including thinking tokens ^[11].

Defaults differ as well. Claude Opus 5.5 and Claude Haiku 5.5 default to medium effort, most other Claude models to high ^[10]; as of October 2026, GPT-6.1 Sol defaults to medium and Grok 4.7 to high, with no way to switch its reasoning off. Comparing "at default" therefore compares different amounts of thinking. Set the effort you would use in production, record output tokens per task and compare cost per completed task. Reasoning models explained goes deeper.

Current base-tier prices of the models in these examples:

To rerun the agent example, open the cost calculator with these volumes and a 90% cached share, or start from your own monthly volumes here:

Estimate the monthly API cost

Input tokens per month

Output tokens per month

Compare costs Open the calculator

Compare capabilities on the same terms

Four rules keep feature comparisons honest:

• Unknown is not no. If no official source documents a value, we show "Unknown" or "No data" rather than guessing. As of October 2026, for example, no official source we found states the maximum output of Grok 4.7. Treat that as a question for the provider, not a missing feature.
• Reasoning comes in grades. Some models have no extended reasoning, some can switch it on or off, and some always reason. Claude Fable 5.1 and Grok 4.7 can't turn it off, while Claude Haiku 5.5 accepts thinking switched off at high effort or below (as of October 2026). On short, simple tasks that shows up in cost and latency.
• Limits differ more than headlines suggest. Context windows listed as "1M" range from 1,000,000 to 1,050,000 tokens, which rarely matters. Maximum output matters more: 65,536 tokens on Gemini 3.1 Pro against 393,216 on DeepSeek-V4.1-Flash. See Context windows explained.
• Release status counts. Gemini 3.1 Pro is listed as a preview and Claude Sonnet 4.5 as deprecated (as of October 2026). A model about to change or retire is a weaker choice, however well it compares today.

Use the compare page

The compare page puts two to four models side by side. Each selection gets a shareable address, such as Claude Sonnet 5.5 vs GPT-6.1 Sol. The table covers:

• Pricing: every price dimension at the base tier, with a "tiered" badge where prompt-length tiers apply. Tool fees and other charges are listed with what they cover, never ranked.
• Specs and capabilities: context window, maximum output, license, release date, availability, knowledge cutoff and the supported inputs, outputs and features.
• Benchmarks and speed: only results that two or more of the models share in the same setting, with self-reported scores marked. The catalog holds no speed results yet (as of October 2026), so the speed rows stay empty for now.
• Best for, strengths and weaknesses: each model's rank in the matching use-case ranking once that ranking is published (related data such as the reasoning capability until then), plus the listed strengths and weaknesses.

All four models below list $2 per 1M input tokens (as of October 2026) but differ in output price ($6 to $12), in where higher long-prompt rates begin (about 200,000 tokens on Grok 4.7 and Gemini 3.1 Pro, above 272,000 on GPT-6.1 Sol, never on Claude Sonnet 5.5), in context window and in release status.

"No data" means we have no verified value; it is never an estimate. Benchmarks are compared only within the same setting.

Read rankings critically

Use-case rankings, such as Best AI for coding, score candidates with a published formula. Before you trust a position, check four things on the ranking page:

1. Weights. The weights table shows how much benchmarks, price, context window, capabilities and editorial scores count. If your priorities differ, so may the order.
2. Relative scores. Each criterion is rescaled to 0–100 across the candidates, so 100 means best among these models, not perfect. Adding or removing a candidate can shift other scores.
3. Coverage. A model's score covers only the criteria it has data for. Models below the coverage threshold (60% unless stated) appear as "Not ranked: insufficient data" instead of being scored with zeros.
4. Date and assumptions. The "Ranking as of" date shows when the snapshot was taken, and the price criterion uses the stated blend ratio. If your ratio differs, recompute the price.

The methodology documents every step. A ranking is a structured shortlist, not a verdict: the last step is still a test on your own inputs.