Multimodal AI: reading documents, images and voice

"Multimodal" means a single model can take in more than text: images, scanned pages, charts, audio and sometimes video. For businesses the value is concrete. Much of what organizations know sits in PDFs, photos, forms and recorded calls, not in clean text fields.

Documents and images

Anthropic's documentation describes sending a PDF and asking about the text, pictures, charts and tables in it. Listed uses include analyzing financial reports, extracting key information from legal documents, translating documents and converting content into structured formats. Because each page is processed as an image alongside its extracted text, dense pages with small fonts, complex tables or heavy graphics can fill the context window before a page limit is reached, and the guidance is to split large documents into sections ^[1].

Practical meaning: a document with tables or charts may not need a separate extraction step before a model can use it, though you should test that on your own files. Limits apply per request, including a 32 MB size cap and a page cap that depends on the model's context window, so check them for your own documents on the vendor's current page ^[1].

Voice and audio

Google's Gemini API documentation lists audio capabilities including transcription and translation, summarization, speaker diarization (labeling who spoke), emotion detection, analysis of specific segments by timestamp, and recognition of non-speech sounds. Audio is counted at 32 tokens per second, so one minute is 1,920 tokens, and a single prompt can hold up to 9.5 hours of audio. Inline requests are limited to 20 MB, with a file upload API for larger files ^[2]. That token rate makes call-recording costs easy to estimate.

Running it locally

Multimodal input is not limited to hosted services. Google's open Gemma 4 family, released April 2 under Apache 2.0, supports image and video input in all four sizes, and audio input (for speech recognition) in the two smallest, which target phones and small devices ^[3].

What to test first

1. Take 50 real documents, including the worst scans, and measure field-level accuracy.

2. For audio, compare transcripts with a human-checked sample, especially names and numbers.

3. Estimate cost with the vendor's token rates for your volume.

4. Decide which content may leave your network, and whether a local model is needed.

For businesses, pilot multimodal AI on one document or audio workflow where manual re-keying is the current cost, and judge it on accuracy against a human-checked sample, not on a demo.