Multimodal AI explained: image, audio, video and file inputs, and what they cost

What "multimodal" means for an AI model

A multimodal model handles more than one kind of data, but input and output are separate abilities. Almost every current text model can look at a photo or a screenshot. Far fewer can listen to a recording or watch a video, and the text models in our catalog do not return images. Anthropic, for example, states that Claude can interpret images but cannot generate or edit them ^[1].

Our catalog therefore records each input and output modality separately. A value of "partial" means the provider documents real restrictions, such as video read as still frames only. You can filter the model directory by image input, audio input, image output or audio output.

Which catalog models support what

As of October 2026, our catalog lists 72 current models (deprecated models are not counted):

• Image input (60 models). Nearly every current Claude, GPT-6, Gemini, Grok, Mistral and Amazon Nova model, plus the image generators, which take reference images for editing. Text-only exceptions include DeepSeek V4 Pro, gpt-oss-120b and gpt-oss-20b, Amazon Nova Micro and most Cohere models.
• Audio input (15 models). All five Gemini 3.x text models and Gemini 4 Argon, Alibaba's Qwen3.8-Omni-Flash, Meta's Muse Wonder, Muse Wonder Full, Muse Spark Open and Muse Spark 1.1 and 1.2 (1.3 only partially), and Amazon's speech models Nova 2 Sonic and Nova 2.5 Sonic. No current Claude, GPT-6, Grok or Mistral model in the catalog accepts audio.
• Video input (22 models, plus 2 with partial support). The Gemini 3.x models and Gemini 4 Argon, Qwen3.7-Plus and most Qwen3.8 models, Amazon Nova Lite, Pro and 2 Lite, the Muse Wonder and Muse Spark models and two Nano Banana models. Not all of them hear the soundtrack: Amazon Nova 2 Lite analyses the visual frames only ^[2], and Alibaba points to Qwen3.8-Omni-Flash when the sound in a video matters ^[3]. The Gemini models and Muse Spark process the audio track as well, though audio on Muse Spark 1.3 is not yet fully supported ^{[4] [5]}.
• Image output (10 models). All are dedicated image models: GPT-Image-2.5 Flare and Sunburst, Nano Banana 2.1, 2 Lite and Pro, Grok Imagine Image and Imagine Image 2.0, Muse Image, Qwen-Image 3.0 Pro and Wan2.7 Image Pro.
• Audio output (2 models). Nova 2 Sonic and Nova 2.5 Sonic, both speech-to-speech models for voice conversations. OpenAI also prices realtime, transcription and text-to-speech models ^[6]; these are not yet in our catalog.

The table compares four text models across modalities.

"No data" means we have no verified value; it is never an estimate. Benchmarks are compared only within the same setting.

How images, audio and video become tokens

Each provider converts media into tokens differently, enough to change your bill.

Images. Anthropic divides an image into 28×28-pixel patches, so a 1000×1000 image costs 1,296 tokens. Claude 4.7 and later models process images up to 2,576 pixels on the long edge (at most 4,784 tokens) and scale larger ones down, which caps the cost; a 4K screenshot can still use about three times as many tokens as on older models ^[1]. OpenAI uses 32-pixel patches and a model-specific multiplier: on GPT-6 Astra a 1024×1024 image at high detail costs 1,229 tokens, and a detail setting trades fidelity for tokens ^[7]. On Gemini 3 models, Google assigns a fixed budget per image through the media_resolution setting: 1,120 tokens by default, 280 at low and 2,240 at ultra_high ^[8]. Amazon bills every image on Nova 2 Lite at a flat 230 tokens ^[2]. Google's general token guide describes yet another rule, 258 tokens per small image or per 768×768 tile ^[9], so count tokens for your own files before estimating costs.

Audio. Google counts audio per second: 25 tokens on Gemini 3 models according to its media resolution guide, 32 according to its audio and token guides ^{[8] [10]}. One hour of audio is therefore 90,000 to 115,200 tokens. A single prompt can hold up to 9.5 hours; audio is downsampled to 16 Kbps and multiple channels are merged into one ^[10].

Video. By default, Gemini samples one frame per second. Google estimates about 100 tokens per second of video at low resolution and about 300 at high resolution, including the soundtrack, so a 1M-token context window holds about three hours or one hour of video respectively ^[4]. For video with a lot of on-screen text, the higher setting matters; for general scenes, Google recommends the low or medium setting ^[8]. On Gemini 3.8 Flash, 3.6 Flash and 3.5 Flash-Lite, an optional agentic mode loads only the parts of a video the prompt needs; Google says it uses up to 88% fewer tokens on long videos, but it can slightly delay the first token on short clips ^[4].

Files and PDFs: one checkbox, four different mechanisms

"File input" on a model page does not tell you what the model actually sees. Provider documentation describes four approaches:

1. Page images plus extracted text. Claude converts each PDF page into an image and supplies the page's text alongside it, so charts and layouts are visible. Anthropic estimates 1,500 to 3,000 text tokens per page plus the image tokens, with up to 600 pages per request (100 when the context window is under 1M tokens) ^[11]. OpenAI does the same for PDFs on vision-capable models ^[12]. Google's document guide counts each PDF page as 258 tokens and does not charge for the extracted text ^[13]; for Gemini 3, its media resolution guide lists 560 tokens per page by default, 280 at low and 1,120 at high ^[8].
2. Text extraction only. For Word documents, presentations and plain text files, OpenAI extracts the text but not embedded images or charts, and parses spreadsheets up to the first 1,000 rows per sheet ^[12]. Gemini also treats non-PDF documents as plain text ^[13].
3. OCR first, then the model. Mistral's Document QnA first runs OCR on the PDF and then passes the result to a language model ^[14].
4. Search instead of reading. When you attach a file to a Grok request, xAI runs a server-side attachment_search tool over it, billed per invocation on top of tokens. The document content does not appear in the message history ^[15].

If your documents contain charts, scanned pages or tables where layout matters, you need the first approach, and OpenAI advises converting other formats to PDF to keep visuals ^[12]. Search-based access suits fact lookups in large files but does not guarantee that the model reads the whole document.

What image and audio input costs

On most text models, image tokens are billed at the normal input rate, so the cost per image is the token count times the input price; a cheap token does not guarantee a cheap image. Using the token counts above and base list prices as of October 2026 (for prompts below any long-context threshold), 1,000 images of roughly 1000×1000 pixels cost:

• Claude Haiku 5.5 ($0.10 per million input tokens): about $0.13 ^[16]
• Gemini 3.8 Flash ($0.75 introductory rate through December 31, 2026): about $0.84, or $1.68 at the $1.50 rate from January 2027 ^[17]
• Claude Sonnet 5.5 ($2): about $2.59 ^[16]
• GPT-6 Astra ($10): about $12.29 ^[6]

The prompt text and the answer are billed on top.

Audio is sometimes priced separately. As of October 2026, Gemini 3.1 Flash-Lite charges $0.25 per million tokens for text, image and video input but $0.50 for audio, while Gemini 3.5 Flash-Lite charges $0.30 for all four ^[17]. At 25 tokens per second, one hour of audio costs about $0.045 on 3.1 Flash-Lite and $0.027 on 3.5 Flash-Lite. An hour of video at roughly 360,000 tokens costs about $0.11 on 3.5 Flash-Lite and $0.27 on Gemini 3.8 Flash. For transcription, the output can cost as much as the audio itself: a 12,000-token transcript adds $0.03 at 3.5 Flash-Lite's $2.50 output rate.

Speech-to-speech models price speech tokens far above text tokens. Amazon lists Nova 2 Sonic at $0.33 / $2.75 per million text input / output tokens, but $3 / $12 for speech input / output ^[18].

The AI cost calculator works with text tokens: convert images or audio into tokens with the rules above, enter them as input tokens and add any separate audio rate by hand.

What image generation costs

Image generators use two billing models:

• Per image. As of October 2026, Muse Image (preview) costs $0.01 per image, and failed or safety-filtered images are not billed ^[19]. Grok Imagine Image costs $0.02. Grok Imagine Image 2.0 costs $0.04 at 1K resolution and low quality, up to $0.08 at 2K and medium quality ^[20]. Qwen-Image 3.0 Pro costs $0.04 at 1K and $0.075 at 2K (Singapore list prices) ^[21].
• Per image token. GPT-Image-2.5 Flare and Sunburst both charge $8 per million image input tokens, $30 per million image output tokens and $5 per million text input tokens ^[6]. The cost per image depends on size and quality: OpenAI points out that equal token rates do not mean equal cost per image and recommends reading the token usage of each response ^[22].

Google publishes both views. Nano Banana 2.1 bills image output at $30 per million tokens, and a 1K image uses 1,120 tokens, or $0.0336; 2K costs $0.0504 and 4K $0.113. Nano Banana Pro costs $0.134 per 1K or 2K image and $0.24 at 4K ^[17].

Editing adds input costs: xAI charges per input image ($0.01 on Imagine Image 2.0) ^[20], Alibaba $0.003 per reference image on Qwen-Image 3.0 Pro ^[21], and OpenAI bills input images as image tokens ^[22]. Compare prices at the resolution and quality you actually need, since the lowest list price often applies only to the smallest size.

Practical use cases and what to check

• Documents, forms and screenshots. Any vision-capable text model can extract data from them. The main levers are resolution and the file mechanism: downsample images when fine detail does not matter, and send charts as PDF pages. For PDFs on Gemini 3, Google notes that OCR quality typically saturates at the medium setting ^[8].
• Long audio and video. The Gemini 3.x text models, Muse Spark (preview) and Qwen3.8-Omni-Flash ^[23] accept both audio and video with a context window of about 1M tokens. For dedicated real-time transcription, Google points to its separate Speech-to-Text service ^[10].
• Voice agents. Speech-to-speech models stream audio in both directions. Nova 2 Sonic uses a bidirectional streaming API, has a default quota of 20 concurrent sessions per account and Region ^[24], and limits each connection to 8 minutes, with a pattern for continuing the conversation ^[25].
• Image generation. Check resolution limits (Nano Banana 2 Lite supports only 1K ^[26]), the number of images per request (1 to 10 on xAI ^[27]) and known weak spots: OpenAI notes that text placement and consistent characters across images can still be unreliable ^[22]. Google adds a SynthID watermark to every generated image ^[26]. For a worked example of generated images, video, voice and 3D models ending up in one publication, see AI images, video, audio and 3D for a digital magazine.

Whatever the modality, check outputs. Anthropic and OpenAI both describe object counts and spatial positions as approximate, and both advise against using their models to interpret medical scans ^{[1] [7]}.

For the general mechanics of token billing, see AI token pricing explained; for how media fills a context window, see context windows explained. To narrow down a model for your task, use how to choose an AI model, the model directory and the compare page. Our methodology explains how the catalog data is collected and verified.